2025-01-04 08:42:55,766 INFO StreamThr :3286844 [internal.py:wandb_internal():85] W&B internal server running at pid: 3286844, started at: 2025-01-04 08:42:55.765722 2025-01-04 08:42:55,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status 2025-01-04 08:42:55,773 INFO WriterThread:3286844 [datastore.py:open_for_write():87] open: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/run-r3b53lzn.wandb 2025-01-04 08:42:55,774 DEBUG SenderThread:3286844 [sender.py:send():379] send: header 2025-01-04 08:42:55,817 DEBUG SenderThread:3286844 [sender.py:send():379] send: run 2025-01-04 08:42:56,043 INFO SenderThread:3286844 [dir_watcher.py:__init__():211] watching files in: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files 2025-01-04 08:42:56,043 INFO SenderThread:3286844 [sender.py:_start_run_threads():1188] run started: r3b53lzn with start time 1735980175.767697 2025-01-04 08:42:56,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: check_version 2025-01-04 08:42:56,068 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: check_version 2025-01-04 08:42:56,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: run_start 2025-01-04 08:42:56,173 DEBUG HandlerThread:3286844 [system_info.py:__init__():26] System info init 2025-01-04 08:42:56,173 DEBUG HandlerThread:3286844 [system_info.py:__init__():41] System info init done 2025-01-04 08:42:56,173 INFO HandlerThread:3286844 [system_monitor.py:start():194] Starting system monitor 2025-01-04 08:42:56,173 INFO SystemMonitor:3286844 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-04 08:42:56,173 INFO HandlerThread:3286844 [system_monitor.py:probe():214] Collecting system info 2025-01-04 08:42:56,174 INFO SystemMonitor:3286844 [interfaces.py:start():188] Started cpu monitoring 2025-01-04 08:42:56,175 INFO SystemMonitor:3286844 [interfaces.py:start():188] Started disk monitoring 2025-01-04 08:42:56,175 INFO SystemMonitor:3286844 [interfaces.py:start():188] Started gpu monitoring 2025-01-04 08:42:56,176 INFO SystemMonitor:3286844 [interfaces.py:start():188] Started memory monitoring 2025-01-04 08:42:56,177 INFO SystemMonitor:3286844 [interfaces.py:start():188] Started network monitoring 2025-01-04 08:42:56,236 DEBUG HandlerThread:3286844 [system_info.py:probe():152] Probing system 2025-01-04 08:42:56,240 DEBUG HandlerThread:3286844 [system_info.py:_probe_git():137] Probing git 2025-01-04 08:42:56,263 DEBUG HandlerThread:3286844 [system_info.py:_probe_git():145] Probing git done 2025-01-04 08:42:56,263 DEBUG HandlerThread:3286844 [system_info.py:probe():200] Probing system done 2025-01-04 08:42:56,263 DEBUG HandlerThread:3286844 [system_monitor.py:probe():223] {'os': 'Linux-6.8.0-48-generic-x86_64-with-glibc2.39', 'python': '3.10.0', 'heartbeatAt': '2025-01-04T08:42:56.237086', 'startedAt': '2025-01-04T08:42:55.751623', 'docker': None, 'cuda': None, 'args': ('--model_family', 'llama', '--apply_instruct_masks', '--token_scaled_loss', '--seq_parallel_size', '8', '--report_to', 'wandb', '--do_train', '--model_name_or_path', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--config_name', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--tokenizer_name', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--run_name', '_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_', '--output_dir', '/scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_', '--config_overrides_json', '', '--gradient_accumulation_steps', '2', '--per_device_train_batch_size', '1', '--bf16', '--learning_rate', '1e-6', '--min_lr_ratio', '0.1', '--lr_scheduler_type', 'cosine', '--max_grad_norm', '1.0', '--adam_beta1', '0.9', '--adam_beta2', '0.95', '--weight_decay', '0.1', '--warmup_ratio', '0.05', '--optim', 'adamw_torch', '--logging_steps', '1', '--log_level', 'info', '--save_steps', '200', '--dataloader_num_workers', '1', '--disable_tqdm', 'true', '--use_fast_tokenizer', 'false', '--remove_unused_columns', 'false', '--ddp_find_unused_parameters', 'false', '--fsdp', 'auto_wrap offload', '--gradient_checkpointing', '--tokenized_mds_train', '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/data/ft/bookclaim_balanced_pack_complete', '--cuda_empty_cache', '--num_train_epochs', '1'), 'state': 'running', 'program': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/prolong-final/finetune.py', 'codePathLocal': 'finetune.py', 'codePath': 'prolong-final/finetune.py', 'git': {'remote': 'https://github.com/chtmp223/BookGen-dev.git', 'commit': '9c36a7f300815c07d8a1bbceedfa1ef3781014f6'}, 'email': 'ctpham@umass.edu', 'root': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev', 'host': 'gpu018', 'username': 'ctpham_umass_edu', 'executable': '/scratch3/workspace/ctpham_umass_edu-ft/envs/prolong-final/bin/python3.10', 'cpu_count': 112, 'cpu_count_logical': 112, 'cpu_freq': {'current': 998.8204107142858, 'min': 800.0, 'max': 3800.0}, 'cpu_freq_per_core': [{'current': 757.247, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.86, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1734.041, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1100.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1200.0, 'min': 800.0, 'max': 3800.0}, {'current': 798.626, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2500.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2900.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2200.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1465.616, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 2751.953, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 788.781, 'min': 800.0, 'max': 3800.0}, {'current': 790.46, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 781.502, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 781.616, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 782.352, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 782.191, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 782.295, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}], 'disk': {'/': {'total': 438.487850189209, 'used': 16.66223907470703}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 2015.3287467956543}} 2025-01-04 08:42:56,263 INFO HandlerThread:3286844 [system_monitor.py:probe():224] Finished collecting system info 2025-01-04 08:42:56,263 INFO HandlerThread:3286844 [system_monitor.py:probe():227] Publishing system info 2025-01-04 08:42:56,264 DEBUG HandlerThread:3286844 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2025-01-04 08:42:57,045 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/conda-environment.yaml 2025-01-04 08:43:02,991 DEBUG HandlerThread:3286844 [system_info.py:_save_conda():224] Saving conda packages done 2025-01-04 08:43:02,995 INFO HandlerThread:3286844 [system_monitor.py:probe():229] Finished publishing system info 2025-01-04 08:43:03,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:03,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 08:43:03,001 DEBUG SenderThread:3286844 [sender.py:send():379] send: files 2025-01-04 08:43:03,001 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-04 08:43:03,046 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/conda-environment.yaml 2025-01-04 08:43:03,047 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-metadata.json 2025-01-04 08:43:03,325 INFO wandb-upload_0:3286844 [upload_job.py:push():130] Uploaded file /tmp/tmpjm5xs8qvwandb/3mt0q14y-wandb-metadata.json 2025-01-04 08:43:03,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: python_packages 2025-01-04 08:43:03,328 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: python_packages 2025-01-04 08:43:03,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:43:03,329 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:43:03,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:03,407 DEBUG SenderThread:3286844 [sender.py:send():379] send: telemetry 2025-01-04 08:43:03,407 DEBUG SenderThread:3286844 [sender.py:send():379] send: config 2025-01-04 08:43:03,409 DEBUG SenderThread:3286844 [sender.py:send():379] send: telemetry 2025-01-04 08:43:03,409 DEBUG SenderThread:3286844 [sender.py:send():379] send: metric 2025-01-04 08:43:03,409 DEBUG SenderThread:3286844 [sender.py:send():379] send: telemetry 2025-01-04 08:43:03,410 DEBUG SenderThread:3286844 [sender.py:send():379] send: metric 2025-01-04 08:43:03,410 WARNING SenderThread:3286844 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2025-01-04 08:43:03,410 DEBUG SenderThread:3286844 [sender.py:send():379] send: telemetry 2025-01-04 08:43:03,410 DEBUG SenderThread:3286844 [sender.py:send():379] send: telemetry 2025-01-04 08:43:03,410 DEBUG SenderThread:3286844 [sender.py:send():379] send: config 2025-01-04 08:43:04,048 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:43:04,048 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/requirements.txt 2025-01-04 08:43:04,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:05,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:06,049 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:43:06,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:06,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:07,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:08,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:09,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:10,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:11,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:12,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:12,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:13,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:14,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:15,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:16,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:17,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:17,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:18,057 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:43:18,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:43:18,328 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:43:18,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:19,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:20,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:21,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:22,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:22,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:23,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:24,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:25,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:26,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:27,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:28,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:28,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:29,064 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/config.yaml 2025-01-04 08:43:29,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:30,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:31,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:32,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:33,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:43:33,329 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:43:33,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:34,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:34,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:35,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:36,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:37,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:38,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:39,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:40,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:40,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:41,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:42,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:43,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:44,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:45,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:46,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:46,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:47,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:48,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:43:48,330 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:43:48,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:49,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:43:49,013 DEBUG SenderThread:3286844 [sender.py:send():379] send: metric 2025-01-04 08:43:49,014 DEBUG SenderThread:3286844 [sender.py:send():379] send: metric 2025-01-04 08:43:49,014 DEBUG SenderThread:3286844 [sender.py:send():379] send: metric 2025-01-04 08:43:49,014 DEBUG SenderThread:3286844 [sender.py:send():379] send: metric 2025-01-04 08:43:49,014 DEBUG SenderThread:3286844 [sender.py:send():379] send: metric 2025-01-04 08:43:49,014 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:43:49,015 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:43:49,017 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:43:49,091 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:43:49,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:50,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:51,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:52,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:52,093 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:43:52,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:53,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:54,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:55,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:56,177 DEBUG SystemMonitor:3286844 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-04 08:43:56,179 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:43:56,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:57,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:43:57,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:58,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:43:59,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:00,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:01,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:02,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:02,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:03,099 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/config.yaml 2025-01-04 08:44:03,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:44:03,330 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:44:03,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:04,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:05,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:06,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:07,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:08,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:08,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:09,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:10,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:11,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:12,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:13,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:14,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:14,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:15,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:16,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:17,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:18,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:44:18,332 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:44:18,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:19,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:19,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:20,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:21,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:44:21,486 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:44:21,486 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:44:21,491 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:44:21,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:22,118 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:44:22,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:23,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:24,120 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:44:24,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:25,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:25,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:26,182 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:44:26,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:27,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:28,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:29,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:30,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:31,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:31,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:32,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:33,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:44:33,333 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:44:33,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:34,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:35,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:36,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:36,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:37,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:38,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:39,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:40,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:41,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:42,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:42,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:43,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:44,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:45,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:46,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:47,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:47,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:48,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:44:48,341 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:44:48,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:49,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:50,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:44:50,216 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:44:50,216 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:44:50,221 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:44:50,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:51,143 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:44:51,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:52,144 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:44:52,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:53,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:53,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:54,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:55,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:56,185 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:44:56,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:57,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:58,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:44:58,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:44:59,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:00,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:01,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:02,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:03,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:45:03,336 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:45:03,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:03,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:04,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:05,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:06,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:07,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:08,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:09,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:09,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:10,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:11,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:12,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:13,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:14,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:14,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:15,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:16,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:17,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:18,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:45:18,338 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:45:18,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:19,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:19,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:19,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:45:19,942 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:45:19,942 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:45:19,945 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:45:20,166 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:45:20,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:21,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:22,167 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:45:22,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:23,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:24,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:25,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:25,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:26,188 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:45:26,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:27,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:28,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:29,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:30,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:31,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:31,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:32,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:33,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:45:33,339 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:45:33,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:34,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:35,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:36,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:36,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:37,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:38,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:39,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:40,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:41,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:42,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:42,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:43,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:44,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:45,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:46,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:47,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:47,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:48,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:45:48,342 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:45:48,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:49,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:50,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:50,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:45:50,898 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:45:50,899 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:45:50,901 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:45:51,193 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:45:51,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:52,194 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:45:52,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:52,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:53,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:54,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:55,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:56,192 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:45:56,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:57,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:58,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:45:58,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:45:59,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:00,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:01,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:02,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:03,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:46:03,343 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:46:03,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:03,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:04,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:05,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:06,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:07,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:08,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:09,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:09,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:10,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:11,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:12,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:13,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:14,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:15,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:15,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:16,687 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:17,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:18,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:46:18,344 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:46:18,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:19,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:20,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:20,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:20,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:46:20,833 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:46:20,833 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:46:20,838 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:46:21,221 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:46:21,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:22,227 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:46:22,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:23,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:24,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:25,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:26,196 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:46:26,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:26,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:27,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:28,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:29,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:30,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:31,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:31,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:32,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:33,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:46:33,345 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:46:33,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:34,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:35,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:36,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:36,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:37,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:38,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:39,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:40,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:41,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:42,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:42,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:43,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:44,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:45,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:46,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:47,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:47,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:48,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:46:48,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:46:48,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:49,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:50,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:51,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:46:51,264 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:46:51,264 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:46:51,269 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:46:51,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:52,255 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:46:52,255 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:46:52,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:53,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:53,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:54,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:55,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:56,200 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:46:56,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:57,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:58,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:46:58,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:46:59,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:00,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:01,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:02,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:03,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:47:03,348 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:47:03,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:03,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:04,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:05,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:06,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:07,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:08,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:09,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:09,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:10,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:11,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:12,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:13,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:14,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:15,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:15,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:16,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:17,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:18,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:47:18,348 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:47:18,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:19,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:20,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:20,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:21,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:47:21,754 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:47:21,755 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:47:21,759 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:47:22,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:22,279 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:47:23,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:24,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:24,280 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:47:25,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:26,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:26,204 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:47:26,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:27,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:28,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:29,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:30,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:31,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:31,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:32,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:33,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:33,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:47:33,348 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:47:34,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:35,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:36,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:36,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:37,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:38,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:39,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:40,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:41,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:42,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:42,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:43,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:44,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:45,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:46,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:47,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:48,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:48,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:47:48,348 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:47:48,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:49,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:50,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:51,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:52,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:53,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:54,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:54,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:47:55,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:56,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:56,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:47:56,197 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:47:56,197 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:47:56,198 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:47:56,205 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:47:56,305 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:47:57,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:58,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:47:58,307 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:47:59,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:00,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:00,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:01,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:02,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:03,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:03,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:48:03,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:48:04,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:05,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:05,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:06,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:07,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:08,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:09,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:10,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:11,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:11,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:12,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:13,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:14,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:15,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:16,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:16,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:17,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:18,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:18,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:48:18,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:48:19,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:20,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:21,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:21,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:22,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:23,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:24,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:25,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:26,216 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:48:26,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:27,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:27,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:27,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:48:27,673 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:48:27,674 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:48:27,675 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:48:28,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:28,330 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:48:28,331 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:48:29,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:30,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:31,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:32,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:32,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:33,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:33,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:48:33,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:48:34,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:35,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:36,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:37,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:37,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:38,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:39,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:40,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:41,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:42,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:43,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:43,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:44,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:45,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:46,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:47,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:48,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:48,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:48:48,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:48:48,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:49,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:50,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:51,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:52,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:53,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:54,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:54,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:48:55,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:55,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:48:55,662 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:48:55,663 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:48:55,664 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:48:56,211 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:48:56,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:56,352 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:48:56,352 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:48:57,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:58,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:59,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:48:59,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:00,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:01,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:02,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:03,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:03,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:49:03,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:49:04,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:04,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:05,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:06,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:07,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:08,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:09,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:10,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:10,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:11,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:12,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:13,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:14,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:15,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:16,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:16,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:17,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:18,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:18,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:49:18,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:49:19,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:20,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:21,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:21,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:22,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:23,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:24,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:25,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:26,218 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:49:26,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:26,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:49:26,806 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:49:26,807 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:49:26,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:26,808 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:49:27,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:27,375 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:49:28,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:28,375 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:49:29,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:30,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:31,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:32,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:32,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:33,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:33,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:49:33,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:49:34,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:35,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:36,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:37,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:37,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:38,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:39,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:40,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:41,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:42,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:43,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:43,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:44,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:45,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:46,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:47,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:48,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:48,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:49:48,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:49:48,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:49,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:50,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:51,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:52,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:53,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:54,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:54,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:49:55,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:56,217 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:49:56,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:56,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:49:56,669 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:49:56,669 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:49:56,671 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:49:57,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:57,402 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:49:58,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:58,403 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:49:59,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:49:59,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:00,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:01,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:02,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:03,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:03,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:50:03,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:50:04,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:04,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:05,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:06,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:07,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:08,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:09,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:10,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:10,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:11,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:12,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:13,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:14,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:15,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:15,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:16,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:17,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:18,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:18,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:50:18,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:50:19,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:20,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:20,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:21,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:22,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:23,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:24,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:25,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:26,218 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:50:26,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:26,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:27,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:28,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:28,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:50:28,866 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:50:28,866 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:50:28,867 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:50:29,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:29,428 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:50:30,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:30,429 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:50:31,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:31,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:32,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:33,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:33,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:50:33,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:50:34,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:35,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:36,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:37,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:37,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:38,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:39,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:40,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:41,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:42,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:43,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:43,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:44,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:45,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:46,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:47,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:48,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:48,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:50:48,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:50:48,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:49,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:50,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:51,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:52,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:53,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:54,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:54,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:50:55,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:56,224 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:50:56,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:57,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:58,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:59,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:50:59,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:00,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:01,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:51:01,042 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:51:01,042 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:51:01,044 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:51:01,454 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:51:01,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:02,455 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:51:02,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:03,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:51:03,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:51:03,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:04,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:04,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:05,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:06,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:07,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:08,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:09,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:10,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:10,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:11,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:12,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:13,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:14,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:15,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:16,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:16,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:17,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:18,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:51:18,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:51:18,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:19,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:20,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:21,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:21,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:22,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:23,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:24,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:25,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:26,235 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:51:26,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:27,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:27,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:28,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:29,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:30,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:31,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:51:31,145 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:51:31,145 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:51:31,146 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:51:31,485 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:51:31,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:32,486 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:51:32,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:33,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:33,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:51:33,358 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:51:33,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:34,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:35,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:36,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:37,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:38,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:38,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:39,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:40,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:41,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:42,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:43,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:43,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:44,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:45,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:46,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:47,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:48,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:51:48,358 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:51:48,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:48,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:49,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:50,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:51,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:52,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:53,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:54,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:54,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:55,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:56,230 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:51:56,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:57,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:58,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:51:59,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:51:59,888 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:00,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:01,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:02,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:02,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:52:02,940 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:52:02,941 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:52:02,944 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:52:03,547 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:52:03,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:52:03,794 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:52:03,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:04,549 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:52:04,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:04,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:05,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:06,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:07,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:08,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:09,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:10,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:10,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:11,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:12,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:13,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:14,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:15,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:16,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:16,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:17,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:18,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:52:18,792 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:52:18,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:19,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:20,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:21,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:21,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:22,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:23,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:24,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:25,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:26,235 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:52:26,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:27,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:27,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:28,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:29,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:30,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:31,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:32,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:32,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:33,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:52:33,482 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:52:33,483 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:52:33,488 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:52:33,571 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:52:33,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:52:33,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:52:33,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:34,571 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:52:34,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:35,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:36,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:37,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:38,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:38,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:39,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:40,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:41,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:42,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:43,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:43,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:44,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:45,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:46,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:47,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:48,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:52:48,888 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:48,888 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:52:48,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:49,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:50,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:51,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:52,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:53,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:54,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:52:54,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:55,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:56,239 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:52:56,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:57,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:58,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:52:59,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:00,888 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:00,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:01,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:02,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:03,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:53:03,887 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:53:03,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:04,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:05,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:06,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:06,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:07,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:08,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:09,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:53:09,078 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:53:09,079 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:53:09,081 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:53:09,604 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:53:09,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:10,605 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:53:10,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:11,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:11,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:12,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:13,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:14,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:15,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:16,888 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:16,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:17,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:18,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:53:18,887 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:53:18,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:19,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:20,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:21,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:22,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:22,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:23,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:24,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:25,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:26,244 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:53:26,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:27,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:27,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:28,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:29,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:30,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:31,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:32,888 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:32,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:33,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:53:33,904 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:53:33,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:34,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:35,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:36,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:37,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:38,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:38,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:39,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:53:39,583 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:53:39,584 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:53:39,589 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:53:39,634 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:53:39,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:40,635 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:53:40,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:41,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:42,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:43,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:43,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:44,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:45,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:46,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:47,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:48,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:53:48,891 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:53:48,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:49,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:49,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:50,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:51,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:52,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:53,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:54,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:53:54,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:55,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:56,247 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:53:56,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:57,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:58,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:53:59,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:00,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:00,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:01,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:02,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:03,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:54:03,891 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:54:03,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:04,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:06,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:06,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:06,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:07,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:08,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:09,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:10,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:11,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:54:11,664 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:54:11,665 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:54:11,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:11,667 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:54:12,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:12,657 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:54:12,658 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:54:13,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:14,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:15,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:16,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:16,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:17,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:18,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:18,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:54:18,891 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:54:19,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:20,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:21,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:22,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:22,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:23,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:24,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:25,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:26,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:26,250 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:54:27,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:27,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:28,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:29,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:30,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:31,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:32,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:32,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:33,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:33,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:54:33,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:54:34,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:35,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:36,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:37,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:37,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:38,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:39,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:40,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:41,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:41,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:54:41,648 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:54:41,648 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:54:41,649 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:54:41,683 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:54:42,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:42,684 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:54:43,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:43,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:44,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:45,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:46,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:47,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:48,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:48,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:54:48,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:54:48,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:49,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:50,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:51,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:52,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:53,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:54,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:54,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:54:55,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:56,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:56,254 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:54:57,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:58,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:59,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:54:59,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:00,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:01,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:02,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:03,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:03,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:55:03,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:55:04,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:04,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:05,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:06,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:07,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:08,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:09,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:10,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:10,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:11,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:12,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:12,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:55:12,102 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:55:12,103 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:55:12,104 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:55:12,713 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:55:12,713 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:55:13,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:14,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:15,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:16,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:16,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:17,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:18,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:18,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:55:18,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:55:19,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:20,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:21,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:22,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:22,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:23,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:24,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:25,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:26,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:26,258 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:55:27,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:27,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:28,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:29,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:30,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:31,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:32,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:32,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:33,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:33,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:55:33,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:55:34,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:35,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:36,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:37,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:38,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:38,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:39,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:40,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:41,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:42,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:42,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:55:42,931 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:55:42,931 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:55:42,934 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:55:43,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:43,738 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:55:43,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:44,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:44,739 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:55:45,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:46,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:47,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:48,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:48,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:55:48,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:55:48,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:49,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:50,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:51,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:52,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:53,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:54,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:54,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:55:55,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:56,262 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:55:56,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:57,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:58,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:59,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:55:59,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:00,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:01,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:02,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:03,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:03,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:56:03,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:56:04,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:05,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:05,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:06,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:07,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:08,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:09,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:10,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:10,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:11,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:12,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:56:12,193 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:56:12,193 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:56:12,194 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:56:12,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:12,766 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:56:12,767 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:56:13,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:14,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:15,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:16,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:16,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:17,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:18,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:18,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:56:18,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:56:19,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:20,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:21,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:21,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:22,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:23,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:24,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:25,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:26,266 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:56:26,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:27,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:27,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:28,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:29,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:30,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:31,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:32,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:32,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:33,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:33,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:56:33,893 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:56:34,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:35,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:36,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:37,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:37,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:38,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:39,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:40,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:40,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:56:40,688 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:56:40,688 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:56:40,689 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:56:40,794 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:56:41,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:42,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:42,796 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:56:43,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:43,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:44,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:45,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:46,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:47,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:48,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:48,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:56:48,893 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:56:48,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:49,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:50,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:51,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:52,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:53,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:54,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:54,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:56:55,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:56,270 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:56:56,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:57,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:58,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:56:59,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:00,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:00,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:01,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:02,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:03,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:03,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:57:03,893 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:57:04,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:05,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:05,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:06,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:07,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:08,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:09,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:10,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:11,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:57:11,028 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:57:11,028 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:57:11,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:11,029 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:57:11,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:11,822 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:57:12,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:13,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:13,823 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:57:14,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:15,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:16,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:16,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:17,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:18,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:18,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:57:18,893 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:57:19,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:20,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:21,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:22,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:22,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:23,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:24,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:25,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:26,274 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:57:26,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:27,276 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:27,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:28,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:29,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:30,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:31,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:32,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:32,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:33,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:33,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:57:33,893 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:57:34,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:35,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:36,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:37,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:38,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:38,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:39,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:40,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:41,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:42,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:43,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:57:43,530 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:57:43,530 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:57:43,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:43,531 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:57:43,846 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:57:43,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:44,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:45,848 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:57:45,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:46,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:47,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:48,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:57:48,893 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:57:48,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:49,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:49,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:50,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:51,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:52,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:53,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:54,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:54,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:55,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:56,278 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:57:56,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:57,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:58,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:57:59,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:57:59,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:00,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:01,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:02,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:03,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:58:03,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:58:03,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:04,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:04,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:05,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:06,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:07,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:08,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:09,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:10,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:10,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:11,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:12,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:13,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:14,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:58:14,280 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:58:14,281 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:58:14,282 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:58:14,879 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:58:14,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:15,880 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:58:15,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:16,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:16,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:17,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:18,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:58:18,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:58:18,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:19,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:20,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:21,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:21,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:22,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:23,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:24,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:25,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:26,282 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:58:26,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:27,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:27,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:28,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:29,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:30,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:31,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:32,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:32,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:33,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:58:33,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:58:33,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:34,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:35,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:36,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:37,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:38,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:38,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:39,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:40,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:41,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:42,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:43,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:58:43,615 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:58:43,616 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:58:43,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:43,617 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:58:43,912 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:58:44,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:45,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:45,914 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:58:46,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:47,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:48,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:48,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:58:48,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:58:49,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:49,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:50,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:51,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:52,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:53,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:54,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:54,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:55,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:56,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:56,287 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:58:57,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:58,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:58:59,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:58:59,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:00,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:01,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:02,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:03,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:03,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:59:03,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:59:04,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:05,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:05,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:06,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:07,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:08,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:09,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:10,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:10,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:11,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:12,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:13,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:13,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:59:13,532 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:59:13,533 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:59:13,534 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:59:13,959 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:59:14,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:15,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:15,960 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:59:16,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:16,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:17,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:18,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:18,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:59:18,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:59:19,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:20,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:21,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:21,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:22,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:23,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:24,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:25,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:26,290 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:59:26,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:26,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:27,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:28,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:29,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:30,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:31,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:32,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:32,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:33,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:33,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:59:33,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:59:34,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:35,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:36,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:37,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:38,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:38,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:39,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:40,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:41,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:42,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:43,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:43,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:43,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 08:59:43,791 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 08:59:43,792 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 08:59:43,793 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 08:59:43,985 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 08:59:44,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:45,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:45,986 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 08:59:46,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:47,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:48,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:48,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:48,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 08:59:48,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 08:59:49,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:50,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:51,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:52,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:53,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:54,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:54,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:55,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:56,294 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 08:59:56,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:57,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:58,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 08:59:59,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 08:59:59,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:00,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:01,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:02,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:03,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:03,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:00:03,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:00:04,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:05,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:05,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:06,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:07,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:08,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:09,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:10,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:10,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:11,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:12,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:12,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:00:12,802 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:00:12,802 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:00:12,803 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:00:13,008 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:00:13,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:14,008 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:00:14,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:15,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:15,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:16,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:17,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:18,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:18,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:00:18,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:00:19,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:20,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:21,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:21,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:22,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:23,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:24,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:25,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:26,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:26,298 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:00:26,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:27,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:28,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:29,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:30,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:31,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:31,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:32,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:33,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:33,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:00:33,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:00:34,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:35,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:36,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:37,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:37,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:38,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:39,385 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:40,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:41,385 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:42,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:42,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:43,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:44,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:00:44,153 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:00:44,153 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:00:44,154 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:00:44,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:45,038 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:00:45,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:46,039 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:00:46,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:47,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:47,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:48,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:48,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:00:48,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:00:49,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:50,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:51,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:52,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:53,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:53,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:54,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:55,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:56,302 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:00:56,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:57,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:58,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:00:58,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:00:59,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:00,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:01,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:02,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:03,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:03,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:01:03,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:01:04,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:04,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:05,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:06,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:07,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:08,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:09,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:09,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:10,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:11,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:12,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:13,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:01:13,108 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:01:13,108 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:01:13,109 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:01:13,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:14,067 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:01:14,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:15,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:15,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:16,068 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:01:16,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:17,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:18,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:18,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:01:18,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:01:19,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:20,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:21,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:21,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:22,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:23,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:24,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:25,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:26,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:26,306 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:01:26,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:27,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:28,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:29,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:30,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:31,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:31,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:32,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:33,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:33,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:01:33,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:01:34,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:35,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:36,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:37,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:37,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:38,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:39,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:40,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:41,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:42,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:42,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:42,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:01:42,913 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:01:42,914 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:01:42,915 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:01:43,096 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:01:43,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:44,097 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:01:44,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:45,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:46,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:47,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:48,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:48,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:48,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:01:48,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:01:49,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:50,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:51,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:52,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:53,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:54,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:54,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:55,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:56,311 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:01:56,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:57,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:58,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:01:59,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:01:59,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:00,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:01,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:02,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:03,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:03,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:02:03,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:02:04,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:05,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:05,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:06,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:07,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:08,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:09,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:10,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:10,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:11,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:12,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:13,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:02:13,395 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:02:13,395 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:02:13,397 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:02:13,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:14,121 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:02:14,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:15,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:15,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:16,123 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:02:16,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:17,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:18,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:18,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:02:18,896 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:02:19,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:20,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:21,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:21,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:22,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:23,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:24,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:25,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:26,314 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:02:26,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:26,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:27,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:28,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:29,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:30,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:31,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:32,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:32,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:33,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:33,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:02:33,896 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:02:34,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:35,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:36,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:37,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:37,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:38,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:39,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:40,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:41,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:41,870 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:02:41,871 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:02:41,871 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:02:41,872 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:02:42,144 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:02:42,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:42,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:43,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:44,145 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:02:44,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:45,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:46,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:47,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:48,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:48,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:48,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:02:48,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:02:49,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:50,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:51,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:52,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:53,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:54,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:02:54,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:55,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:56,317 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:02:56,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:57,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:58,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:02:59,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:00,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:00,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:01,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:02,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:03,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:03,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:03:03,910 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:03:04,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:05,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:05,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:06,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:07,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:08,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:09,703 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:03:09,704 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:03:09,705 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:03:09,707 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:03:10,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:10,167 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:03:10,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:10,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:11,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:12,168 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:03:12,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:13,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:14,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:15,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:16,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:16,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:17,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:18,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:18,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:03:18,898 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:03:19,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:20,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:21,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:22,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:22,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:23,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:24,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:25,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:26,321 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:03:26,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:27,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:27,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:28,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:29,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:30,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:31,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:32,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:33,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:33,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:33,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:03:33,898 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:03:34,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:35,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:36,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:37,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:38,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:38,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:39,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:40,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:41,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:42,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:03:42,733 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:03:42,733 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:03:42,734 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:03:43,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:43,196 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:03:43,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:44,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:44,196 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:03:45,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:46,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:47,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:48,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:48,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:03:48,898 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:03:48,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:49,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:50,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:51,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:52,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:53,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:54,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:54,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:03:55,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:56,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:56,325 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:03:57,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:58,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:59,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:03:59,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:00,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:01,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:02,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:03,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:03,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:04:03,899 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:04:04,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:05,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:05,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:06,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:07,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:08,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:09,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:10,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:10,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:11,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:12,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:13,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:13,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:04:13,791 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:04:13,791 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:04:13,793 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:04:14,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:14,232 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:04:15,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:15,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:16,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:16,235 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:04:17,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:18,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:18,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:04:18,899 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:04:19,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:20,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:21,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:21,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:22,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:23,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:24,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:25,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:26,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:26,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:26,330 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:04:27,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:28,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:29,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:30,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:31,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:31,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:32,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:33,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:33,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:04:33,899 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:04:34,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:35,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:36,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:37,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:37,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:38,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:39,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:40,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:41,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:42,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:42,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:43,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:44,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:44,788 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:04:44,789 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:04:44,790 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:04:44,791 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:04:45,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:45,270 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:04:46,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:46,271 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:04:47,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:47,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:48,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:48,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:04:48,899 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:04:49,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:50,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:51,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:52,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:53,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:53,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:54,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:55,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:56,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:56,332 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:04:57,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:58,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:04:58,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:04:59,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:00,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:01,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:02,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:03,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:03,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:05:03,900 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:05:03,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:04,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:05,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:06,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:07,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:08,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:09,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:09,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:10,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:11,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:12,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:13,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:14,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:14,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:15,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:16,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:16,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:05:16,785 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:05:16,785 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:05:16,786 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:05:17,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:17,299 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:05:18,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:18,300 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:05:18,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:05:18,899 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:05:19,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:20,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:20,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:21,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:22,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:23,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:24,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:25,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:25,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:26,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:26,336 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:05:27,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:28,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:29,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:30,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:30,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:31,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:32,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:33,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:33,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:05:33,899 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:05:34,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:35,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:36,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:36,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:37,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:38,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:39,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:40,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:41,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:41,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:42,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:43,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:44,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:45,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:46,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:46,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:46,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:05:46,493 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:05:46,493 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:05:46,494 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:05:47,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:47,324 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:05:48,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:48,325 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:05:48,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:05:48,901 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:05:49,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:50,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:51,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:52,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:52,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:53,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:54,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:55,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:56,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:56,340 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:05:57,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:57,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:05:58,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:05:59,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:00,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:01,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:02,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:03,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:03,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:03,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:06:03,900 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:06:04,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:05,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:06,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:07,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:08,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:09,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:09,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:10,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:11,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:12,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:13,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:14,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:14,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:15,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:16,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:06:16,005 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:06:16,005 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:06:16,006 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:06:16,350 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:06:16,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:17,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:18,351 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:06:18,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:18,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:06:18,900 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:06:19,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:20,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:20,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:21,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:22,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:23,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:24,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:25,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:25,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:26,344 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:06:26,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:27,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:28,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:29,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:30,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:31,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:31,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:32,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:33,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:33,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:06:33,900 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:06:34,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:35,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:36,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:37,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:37,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:38,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:39,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:40,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:41,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:42,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:43,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:43,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:44,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:45,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:46,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:47,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:48,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:48,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:48,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:06:48,907 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:06:49,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:49,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:06:49,951 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:06:49,951 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:06:49,953 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:06:50,381 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:06:50,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:51,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:52,382 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:06:52,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:53,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:53,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:54,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:55,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:56,348 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:06:56,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:57,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:58,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:06:59,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:06:59,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:00,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:01,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:02,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:03,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:03,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:07:03,904 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:07:04,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:04,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:05,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:06,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:07,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:08,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:09,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:09,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:10,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:11,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:12,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:13,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:14,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:14,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:15,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:16,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:17,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:18,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:18,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:07:18,904 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:07:19,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:07:19,251 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:07:19,251 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:07:19,252 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:07:19,408 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:07:19,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:19,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:20,409 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:07:20,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:21,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:22,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:23,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:24,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:25,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:25,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:26,352 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:07:26,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:27,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:28,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:29,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:30,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:31,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:31,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:32,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:33,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:33,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:07:33,904 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:07:34,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:35,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:36,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:36,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:37,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:38,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:39,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:40,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:41,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:41,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:42,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:43,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:44,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:45,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:46,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:46,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:47,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:48,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:48,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:07:48,904 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:07:49,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:50,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:07:50,050 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:07:50,050 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:07:50,051 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:07:50,438 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:07:50,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:51,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:52,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:52,440 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:07:52,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:53,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:54,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:55,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:56,356 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:07:56,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:57,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:07:57,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:58,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:07:59,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:00,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:01,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:02,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:02,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:03,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:03,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:08:03,974 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:08:04,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:05,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:06,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:07,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:07,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:08,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:09,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:10,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:11,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:12,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:13,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:13,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:14,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:15,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:16,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:17,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:18,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:18,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:18,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:08:18,980 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:08:19,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:20,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:08:20,621 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:08:20,621 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:08:20,622 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:08:21,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:21,466 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:08:22,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:22,467 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:08:23,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:23,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:24,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:25,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:26,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:26,360 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:08:27,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:28,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:29,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:29,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:30,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:31,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:32,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:33,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:33,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:08:33,976 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:08:34,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:34,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:35,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:36,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:37,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:38,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:39,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:39,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:40,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:41,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:42,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:43,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:44,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:44,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:45,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:46,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:47,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:48,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:48,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:08:48,811 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:08:48,812 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:08:48,814 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:08:49,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:49,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:08:49,219 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:08:49,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:49,488 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:08:50,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:50,489 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:08:51,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:52,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:53,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:54,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:54,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:08:55,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:56,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:56,364 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:08:57,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:58,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:08:59,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:00,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:00,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:01,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:02,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:03,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:04,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:09:04,214 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:09:04,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:05,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:05,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:06,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:07,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:08,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:09,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:10,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:10,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:11,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:12,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:13,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:14,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:15,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:15,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:16,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:17,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:18,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:19,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:09:19,227 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:09:19,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:20,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:20,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:09:20,560 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:09:20,560 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:09:20,562 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:09:20,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:21,513 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:09:21,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:22,514 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:09:22,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:23,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:24,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:25,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:26,368 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:09:26,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:26,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:27,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:28,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:29,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:30,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:31,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:31,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:32,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:33,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:34,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:09:34,218 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:09:34,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:35,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:36,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:36,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:37,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:38,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:39,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:40,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:41,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:41,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:42,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:43,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:44,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:45,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:46,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:46,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:47,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:48,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:49,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:09:49,220 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:09:49,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:50,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:09:50,164 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:09:50,164 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:09:50,165 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:09:50,541 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:09:50,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:51,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:52,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:52,543 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:09:52,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:53,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:54,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:55,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:56,374 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:09:56,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:57,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:09:57,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:58,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:09:59,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:00,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:01,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:02,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:02,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:03,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:04,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:10:04,220 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:10:04,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:05,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:06,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:07,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:08,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:08,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:09,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:10,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:11,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:12,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:13,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:14,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:14,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:15,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:16,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:17,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:18,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:19,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:10:19,224 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:10:19,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:20,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:10:20,198 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:10:20,199 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:10:20,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:20,200 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:10:20,568 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:10:20,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:21,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:22,569 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:10:22,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:23,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:24,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:25,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:25,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:26,376 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:10:26,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:27,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:28,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:29,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:30,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:31,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:31,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:32,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:33,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:34,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:10:34,224 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:10:34,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:35,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:36,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:37,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:37,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:38,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:39,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:40,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:41,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:42,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:42,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:43,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:44,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:45,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:46,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:47,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:47,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:48,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:49,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:10:49,224 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:10:49,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:50,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:10:50,135 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:10:50,136 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:10:50,137 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:10:50,602 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:10:50,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:51,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:52,603 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:10:52,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:53,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:53,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:54,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:55,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:56,381 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:10:56,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:57,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:58,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:10:58,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:10:59,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:00,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:01,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:02,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:03,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:03,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:04,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:11:04,224 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:11:05,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:06,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:07,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:08,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:09,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:09,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:10,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:11,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:12,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:13,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:14,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:15,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:15,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:16,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:17,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:18,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:19,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:19,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:11:19,232 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:11:20,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:20,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:21,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:22,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:22,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:11:22,541 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:11:22,542 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:11:22,543 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:11:22,632 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:11:23,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:24,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:24,633 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:11:25,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:26,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:26,385 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:11:26,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:27,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:28,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:29,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:30,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:31,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:31,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:32,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:33,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:34,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:34,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:11:34,226 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:11:35,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:36,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:37,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:37,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:38,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:39,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:40,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:41,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:42,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:42,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:43,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:44,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:45,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:46,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:47,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:47,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:48,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:49,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:49,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:11:49,236 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:11:50,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:51,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:52,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:52,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:11:52,417 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:11:52,417 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:11:52,419 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:11:52,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:52,658 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:11:53,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:54,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:54,659 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:11:55,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:56,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:56,389 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:11:57,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:58,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:11:58,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:11:59,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:00,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:01,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:02,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:03,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:03,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:04,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:04,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:12:04,230 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:12:05,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:06,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:07,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:08,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:09,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:09,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:10,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:11,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:12,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:13,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:14,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:15,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:15,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:16,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:17,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:18,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:19,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:19,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:12:19,237 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:12:20,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:20,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:21,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:22,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:22,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:12:22,633 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:12:22,634 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:12:22,635 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:12:22,682 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:12:23,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:24,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:24,683 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:12:25,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:26,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:26,392 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:12:26,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:27,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:28,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:29,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:30,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:31,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:31,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:32,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:33,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:34,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:34,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:12:34,234 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:12:35,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:36,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:36,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:37,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:38,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:39,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:40,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:41,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:41,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:42,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:43,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:44,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:45,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:46,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:46,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:47,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:48,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:49,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:49,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:12:49,243 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:12:50,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:51,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:51,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:52,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:52,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:12:52,299 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:12:52,299 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:12:52,300 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:12:52,718 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:12:53,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:54,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:54,719 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:12:55,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:56,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:56,396 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:12:57,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:57,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:12:58,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:12:59,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:00,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:01,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:02,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:02,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:03,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:04,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:04,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:13:04,241 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:13:05,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:06,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:07,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:08,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:08,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:09,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:10,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:11,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:12,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:13,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:13,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:14,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:15,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:16,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:17,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:18,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:18,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:19,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:19,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:13:19,246 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:13:20,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:21,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:22,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:23,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:23,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:13:23,241 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:13:23,242 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:13:23,243 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:13:23,743 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:13:24,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:24,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:24,744 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:13:25,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:26,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:26,400 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:13:27,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:28,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:29,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:29,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:30,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:31,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:32,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:33,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:34,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:34,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:13:34,242 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:13:34,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:35,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:36,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:37,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:38,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:39,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:39,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:40,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:41,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:42,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:43,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:44,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:45,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:45,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:46,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:47,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:48,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:49,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:49,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:13:49,248 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:13:50,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:50,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:51,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:52,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:52,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:13:52,519 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:13:52,520 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:13:52,521 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:13:52,773 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:13:53,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:54,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:54,774 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:13:55,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:56,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:56,403 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:13:56,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:13:57,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:58,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:13:59,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:00,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:01,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:01,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:02,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:03,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:04,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:04,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:14:04,245 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:14:05,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:06,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:07,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:07,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:08,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:09,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:10,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:11,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:12,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:12,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:13,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:14,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:15,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:16,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:17,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:18,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:18,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:19,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:19,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:14:19,254 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:14:20,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:21,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:22,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:23,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:23,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:24,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:14:24,217 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:14:24,217 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:14:24,218 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:14:24,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:24,805 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:14:24,805 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:14:25,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:26,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:26,407 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:14:27,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:28,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:29,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:29,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:30,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:31,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:32,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:33,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:34,248 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:14:34,248 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:14:34,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:34,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:35,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:36,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:37,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:38,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:39,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:39,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:40,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:41,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:42,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:43,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:44,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:45,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:45,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:46,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:47,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:48,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:49,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:14:49,251 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:14:49,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:50,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:50,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:51,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:52,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:53,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:53,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:14:53,566 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:14:53,566 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:14:53,568 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:14:53,830 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:14:54,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:54,831 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:14:55,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:55,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:14:56,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:56,410 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:14:57,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:58,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:14:59,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:00,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:01,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:01,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:02,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:03,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:04,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:15:04,251 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:15:04,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:05,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:06,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:06,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:07,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:08,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:09,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:10,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:11,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:11,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:12,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:13,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:14,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:15,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:16,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:17,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:17,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:18,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:19,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:15:19,259 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:15:19,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:20,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:21,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:22,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:22,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:23,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:24,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:24,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:15:24,414 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:15:24,415 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:15:24,416 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:15:24,856 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:15:24,857 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:15:25,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:26,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:26,414 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:15:27,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:28,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:28,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:29,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:30,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:31,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:32,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:33,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:33,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:34,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:15:34,255 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:15:34,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:35,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:36,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:37,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:38,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:39,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:39,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:40,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:41,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:42,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:43,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:44,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:45,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:45,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:46,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:47,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:48,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:49,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:15:49,259 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:15:49,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:50,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:50,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:51,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:52,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:53,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:53,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:15:53,807 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:15:53,807 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:15:53,808 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:15:53,882 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:15:54,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:54,883 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:15:55,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:56,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:56,417 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:15:56,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:15:57,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:58,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:15:59,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:00,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:01,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:01,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:02,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:03,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:04,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:16:04,256 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:16:04,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:05,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:06,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:07,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:07,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:08,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:09,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:10,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:11,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:12,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:12,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:13,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:14,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:15,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:16,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:17,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:18,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:18,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:19,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:16:19,260 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:16:19,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:20,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:21,385 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:22,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:23,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:23,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:24,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:24,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:16:24,621 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:16:24,622 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:16:24,624 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:16:24,914 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:16:24,914 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:16:25,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:26,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:26,422 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:16:27,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:28,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:29,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:29,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:30,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:31,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:32,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:33,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:34,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:16:34,257 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:16:34,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:35,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:35,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:36,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:37,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:38,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:39,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:40,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:40,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:41,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:42,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:43,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:44,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:45,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:46,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:46,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:47,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:48,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:49,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:16:49,257 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:16:49,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:50,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:51,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:51,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:52,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:53,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:54,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:55,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:56,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:16:56,426 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:16:56,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:57,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:16:57,124 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:16:57,124 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:16:57,126 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:16:57,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:57,945 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:16:58,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:16:58,946 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:16:59,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:00,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:01,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:02,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:02,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:03,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:04,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:17:04,257 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:17:04,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:05,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:06,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:07,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:08,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:08,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:09,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:10,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:11,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:12,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:13,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:13,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:14,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:15,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:16,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:17,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:18,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:19,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:19,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:17:19,269 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:17:19,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:20,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:21,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:22,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:23,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:24,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:24,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:17:24,406 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:17:24,406 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:17:24,408 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:17:24,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:24,973 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:17:24,973 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:17:25,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:26,431 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:17:26,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:27,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:28,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:29,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:29,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:30,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:31,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:32,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:33,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:34,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:17:34,261 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:17:34,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:35,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:35,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:36,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:37,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:38,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:39,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:40,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:40,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:41,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:42,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:43,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:44,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:45,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:46,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:46,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:47,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:48,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:49,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:17:49,261 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:17:49,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:50,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:51,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:51,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:52,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:53,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:54,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:55,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:56,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:17:56,432 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:17:56,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:17:56,745 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:17:56,745 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:17:56,746 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:17:57,010 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:17:57,011 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:17:57,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:58,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:17:59,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:00,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:01,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:02,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:02,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:03,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:04,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:04,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:18:04,261 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:18:05,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:06,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:07,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:08,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:08,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:09,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:10,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:11,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:12,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:13,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:13,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:14,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:15,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:16,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:17,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:18,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:19,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:19,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:19,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:18:19,261 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:18:20,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:21,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:22,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:23,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:24,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:24,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:25,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:26,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:26,434 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:18:26,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:18:26,473 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:18:26,473 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:18:26,475 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:18:27,044 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:18:27,044 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:18:27,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:28,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:29,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:29,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:30,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:31,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:32,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:33,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:34,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:34,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:18:34,262 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:18:35,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:35,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:36,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:37,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:38,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:39,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:40,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:40,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:41,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:42,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:43,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:44,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:45,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:46,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:46,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:47,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:48,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:49,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:18:49,265 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:18:49,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:50,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:51,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:51,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:52,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:53,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:54,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:54,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:18:54,410 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:18:54,410 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:18:54,411 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:18:55,070 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:18:55,070 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:18:55,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:56,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:56,438 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:18:56,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:18:57,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:58,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:18:59,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:00,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:01,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:02,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:02,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:03,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:04,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:19:04,262 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:19:04,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:05,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:06,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:07,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:08,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:08,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:09,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:10,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:11,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:12,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:13,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:13,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:14,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:15,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:16,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:17,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:18,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:18,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:19,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:19:19,269 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:19:19,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:20,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:21,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:21,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:19:21,748 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:19:21,748 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:19:21,749 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:19:22,099 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:19:22,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:23,100 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:19:23,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:23,788 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:24,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:25,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:26,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:26,442 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:19:27,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:28,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:29,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:29,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:30,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:31,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:32,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:33,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:34,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:19:34,494 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:19:34,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:34,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:35,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:36,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:37,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:38,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:39,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:40,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:40,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:41,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:42,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:43,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:44,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:45,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:45,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:46,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:47,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:48,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:49,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:49,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:19:49,495 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:19:50,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:50,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:51,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:19:51,163 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:19:51,163 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:19:51,165 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:19:51,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:52,128 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:19:52,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:53,128 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:19:53,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:54,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:55,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:56,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:19:56,446 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:19:56,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:57,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:58,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:19:59,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:00,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:01,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:02,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:02,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:03,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:04,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:20:04,496 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:20:04,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:05,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:06,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:07,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:07,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:08,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:09,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:10,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:11,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:12,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:12,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:13,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:14,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:15,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:16,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:17,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:17,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:18,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:19,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:20:19,502 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:20:19,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:20,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:21,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:22,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:22,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:20:22,212 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:20:22,212 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:20:22,214 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:20:22,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:23,163 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:20:23,163 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:20:23,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:24,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:25,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:26,449 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:20:26,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:27,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:27,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:28,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:29,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:30,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:31,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:32,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:33,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:33,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:34,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:20:34,501 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:20:34,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:35,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:36,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:37,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:38,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:38,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:39,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:40,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:41,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:42,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:43,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:43,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:44,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:45,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:46,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:47,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:48,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:48,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:49,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:20:49,505 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:20:49,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:50,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:51,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:52,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:53,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:53,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:20:53,260 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:20:53,261 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:20:53,262 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:20:53,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:54,190 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:20:54,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:55,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:56,192 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:20:56,453 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:20:56,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:57,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:58,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:20:58,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:20:59,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:00,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:01,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:02,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:03,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:04,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:04,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:21:04,505 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:21:04,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:05,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:06,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:07,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:08,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:09,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:10,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:10,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:11,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:12,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:13,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:14,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:15,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:16,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:16,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:17,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:18,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:19,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:21:19,509 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:21:19,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:20,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:21,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:21,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:22,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:22,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:21:22,772 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:21:22,773 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:21:22,774 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:21:23,217 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:21:23,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:24,218 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:21:24,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:25,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:26,457 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:21:26,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:26,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:27,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:28,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:29,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:30,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:31,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:32,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:32,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:33,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:34,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:21:34,507 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:21:34,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:35,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:36,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:37,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:38,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:38,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:39,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:40,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:41,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:42,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:43,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:43,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:44,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:45,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:46,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:47,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:48,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:48,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:49,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:21:49,513 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:21:49,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:50,687 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:51,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:52,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:53,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:53,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:21:53,370 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:21:53,370 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:21:53,371 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:21:53,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:54,244 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:21:54,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:55,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:56,246 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:21:56,461 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:21:56,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:57,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:58,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:21:58,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:21:59,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:00,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:01,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:02,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:03,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:04,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:04,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:22:04,513 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:22:04,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:05,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:06,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:07,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:08,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:09,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:10,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:10,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:11,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:12,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:13,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:14,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:15,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:16,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:16,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:17,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:18,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:19,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:22:19,521 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:22:19,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:20,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:21,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:21,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:22,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:22:22,637 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:22:22,637 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:22:22,638 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:22:23,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:23,271 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:22:24,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:24,272 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:22:25,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:26,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:26,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:26,465 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:22:27,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:28,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:29,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:30,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:31,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:32,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:32,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:33,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:34,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:34,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:22:34,516 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:22:35,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:36,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:37,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:37,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:38,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:39,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:40,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:41,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:42,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:43,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:43,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:44,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:45,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:46,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:47,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:48,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:48,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:49,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:49,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:22:49,517 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:22:50,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:51,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:52,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:52,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:22:52,255 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:22:52,256 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:22:52,257 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:22:52,297 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:22:53,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:53,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:54,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:54,298 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:22:55,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:56,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:56,469 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:22:57,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:58,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:22:58,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:22:59,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:00,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:01,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:02,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:03,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:04,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:04,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:04,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:23:04,516 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:23:05,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:06,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:07,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:08,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:09,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:09,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:10,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:11,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:12,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:13,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:14,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:14,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:15,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:16,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:17,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:18,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:19,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:19,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:19,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:23:19,535 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:23:20,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:21,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:22,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:23,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:24,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:24,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:25,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:25,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:23:25,136 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:23:25,137 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:23:25,138 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:23:25,331 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:23:26,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:26,332 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:23:26,474 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:23:27,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:28,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:29,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:29,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:30,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:31,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:32,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:33,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:34,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:34,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:23:34,521 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:23:34,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:35,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:36,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:37,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:38,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:39,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:40,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:40,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:41,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:42,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:43,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:44,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:45,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:46,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:46,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:47,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:48,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:49,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:49,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:23:49,521 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:23:50,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:51,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:52,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:52,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:53,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:54,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:55,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:56,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:56,476 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:23:56,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:23:56,900 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:23:56,900 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:23:56,901 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:23:57,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:57,357 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:23:57,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:23:58,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:23:58,358 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:23:59,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:00,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:01,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:02,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:03,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:03,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:04,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:04,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:24:04,521 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:24:05,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:06,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:07,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:08,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:09,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:09,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:10,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:11,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:12,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:13,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:14,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:14,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:15,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:16,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:17,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:18,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:19,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:19,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:24:19,522 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:24:19,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:20,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:21,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:22,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:23,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:24,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:25,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:25,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:26,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:26,486 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:24:27,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:27,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:24:27,922 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:24:27,922 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:24:27,924 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:24:28,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:28,388 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:24:29,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:30,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:30,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:30,389 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:24:31,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:32,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:33,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:34,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:34,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:24:34,522 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:24:35,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:36,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:36,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:37,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:38,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:39,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:40,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:41,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:41,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:42,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:43,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:44,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:45,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:46,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:47,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:47,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:48,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:49,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:49,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:24:49,522 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:24:50,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:51,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:52,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:52,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:53,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:54,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:55,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:56,484 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:24:56,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:56,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:24:56,939 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:24:56,940 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:24:56,941 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:24:57,429 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:24:57,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:57,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:24:58,430 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:24:58,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:24:59,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:00,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:01,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:02,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:03,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:03,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:04,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:04,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:25:04,522 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:25:05,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:06,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:07,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:08,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:09,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:09,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:10,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:11,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:12,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:13,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:14,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:15,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:15,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:16,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:17,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:18,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:19,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:19,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:25:19,522 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:25:20,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:21,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:21,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:22,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:23,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:24,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:25,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:26,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:26,500 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:25:26,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:27,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:28,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:28,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:25:28,624 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:25:28,624 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:25:28,626 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:25:29,452 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:25:29,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:30,453 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:25:30,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:31,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:32,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:32,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:33,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:34,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:34,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:25:34,522 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:25:35,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:36,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:37,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:38,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:38,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:39,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:40,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:41,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:42,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:43,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:44,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:44,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:45,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:46,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:47,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:48,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:49,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:49,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:25:49,522 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:25:49,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:50,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:51,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:52,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:53,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:54,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:54,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:55,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:56,508 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:25:56,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:57,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:58,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:25:59,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:25:59,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:00,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:26:00,314 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:26:00,315 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:26:00,316 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:26:00,489 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:26:00,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:01,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:02,490 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:26:02,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:03,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:04,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:26:04,523 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:26:04,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:04,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:05,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:06,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:07,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:08,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:09,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:10,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:10,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:11,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:12,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:13,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:14,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:15,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:15,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:16,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:17,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:18,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:19,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:26:19,523 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:26:19,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:20,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:21,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:21,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:22,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:23,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:24,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:25,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:26,517 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:26:26,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:26,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:27,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:28,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:29,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:30,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:31,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:32,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:26:32,067 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:26:32,067 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:26:32,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:32,068 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:26:32,520 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:26:33,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:34,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:34,522 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:26:34,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:26:34,523 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:26:35,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:36,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:37,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:37,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:38,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:39,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:40,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:41,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:42,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:43,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:43,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:44,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:45,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:46,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:47,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:48,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:48,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:49,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:49,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:26:49,523 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:26:50,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:51,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:52,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:53,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:54,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:26:54,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:55,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:56,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:56,511 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:26:57,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:58,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:59,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:26:59,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:00,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:01,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:01,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:27:01,611 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:27:01,611 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:27:01,613 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:27:02,543 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:27:02,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:03,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:04,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:27:04,523 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:27:04,545 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:27:04,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:04,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:05,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:06,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:07,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:08,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:09,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:10,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:10,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:11,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:12,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:13,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:14,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:15,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:16,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:16,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:17,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:18,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:19,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:27:19,523 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:27:19,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:20,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:21,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:22,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:22,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:23,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:24,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:25,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:26,512 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:27:26,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:27,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:27,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:28,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:29,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:30,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:31,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:32,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:33,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:33,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:27:33,435 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:27:33,436 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:27:33,441 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:27:33,568 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:27:33,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:34,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:27:34,524 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:27:34,570 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:27:34,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:35,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:36,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:37,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:38,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:38,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:39,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:40,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:41,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:42,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:43,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:43,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:44,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:45,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:46,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:47,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:48,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:49,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:27:49,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 09:27:49,523 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: stop_status 2025-01-04 09:27:49,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:50,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:51,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:52,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:53,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:54,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:55,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:56,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:57,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:58,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:27:59,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:00,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:01,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:28:02,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:03,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:04,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:05,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:06,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:07,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:08,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:09,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:10,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:10,906 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:28:10,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:10,908 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:28:10,909 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:28:10,910 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:28:11,598 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:28:11,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:12,599 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:28:12,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:13,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:14,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:15,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:15,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:16,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:17,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:18,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:19,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:20,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:20,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:21,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:22,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:23,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:24,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:25,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:25,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:26,530 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:28:26,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:27,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:28,781 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:29,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:30,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:31,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:31,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:31,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:28:31,850 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:28:31,850 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:28:31,851 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:28:32,624 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:28:33,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:34,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:34,626 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:28:35,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:36,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:36,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:37,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:38,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:39,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:40,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:41,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:41,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:42,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:43,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:44,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:45,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:46,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:46,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:47,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:48,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:49,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:50,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:51,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:51,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:52,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:53,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:54,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:55,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:56,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:56,524 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:28:57,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:57,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:28:58,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:28:59,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:00,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:01,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:02,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:02,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:03,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:04,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:04,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:29:04,407 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:29:04,407 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:29:04,412 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:29:04,665 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:29:05,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:06,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:06,667 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:29:07,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:08,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:08,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:09,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:10,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:11,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:12,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:13,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:13,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:14,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:15,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:16,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:17,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:18,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:18,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:19,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:20,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:21,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:22,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:23,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:23,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:24,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:25,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:26,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:26,528 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:29:27,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:28,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:28,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:29,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:30,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:31,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:31,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:29:31,535 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:29:31,536 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:29:31,537 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:29:31,690 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:29:32,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:32,691 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:29:33,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:33,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:34,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:35,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:36,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:37,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:38,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:38,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:39,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:40,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:41,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:42,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:43,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:43,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:44,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:45,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:46,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:47,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:48,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:48,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:49,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:50,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:51,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:52,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:53,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:53,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:54,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:55,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:56,530 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:29:56,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:57,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:58,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:29:59,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:29:59,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:00,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:01,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:02,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:03,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:30:03,693 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:30:03,693 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:30:03,695 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:30:03,712 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:30:04,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:04,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:04,713 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:30:04,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:05,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:06,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:07,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:08,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:09,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:09,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:10,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:11,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:12,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:13,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:14,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:14,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:15,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:16,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:17,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:18,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:19,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:19,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:20,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:21,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:22,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:23,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:24,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:24,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:25,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:26,534 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:30:26,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:27,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:28,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:29,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:30,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:30,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:31,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:32,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:33,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:34,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:30:34,669 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:30:34,669 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:30:34,670 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:30:34,746 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:30:35,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:35,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:35,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:36,748 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:30:36,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:37,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:38,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:39,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:40,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:40,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:41,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:42,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:43,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:44,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:45,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:45,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:46,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:47,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:48,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:49,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:50,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:50,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:51,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:52,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:53,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:54,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:55,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:30:55,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:56,538 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:30:56,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:57,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:58,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:30:59,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:00,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:01,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:01,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:02,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:03,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:04,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:05,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:06,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:31:06,024 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:31:06,025 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:31:06,026 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:31:06,778 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:31:06,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:07,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:07,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:08,780 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:31:08,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:09,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:10,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:11,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:12,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:12,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:13,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:14,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:15,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:16,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:17,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:17,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:18,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:20,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:20,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:21,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:22,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:22,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:23,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:24,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:25,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:26,543 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:31:26,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:27,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:27,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:28,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:29,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:30,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:31,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:32,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:32,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:33,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:34,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:35,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:36,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:37,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:37,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:38,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:39,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:31:39,313 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:31:39,314 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:31:39,315 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:31:39,808 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:31:39,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:40,809 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:31:40,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:41,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:42,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:43,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:43,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:44,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:45,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:46,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:47,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:48,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:48,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:49,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:50,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:51,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:52,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:53,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:53,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:54,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:55,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:56,546 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:31:56,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:57,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:58,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:31:58,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:31:59,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:00,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:01,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:02,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:03,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:03,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:04,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:05,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:06,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:07,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:08,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:08,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:32:08,635 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:32:08,636 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:32:08,637 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:32:08,834 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:32:09,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:10,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:10,836 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:32:11,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:12,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:13,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:13,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:14,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:15,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:16,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:17,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:18,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:18,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:19,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:20,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:21,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:22,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:23,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:23,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:24,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:25,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:26,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:26,550 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:32:27,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:28,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:29,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:29,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:30,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:31,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:32,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:33,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:34,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:34,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:35,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:36,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:37,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:38,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:39,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:39,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:39,868 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:32:39,870 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:32:39,871 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:32:39,872 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:32:40,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:40,866 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:32:40,866 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:32:41,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:42,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:43,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:44,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:44,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:45,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:46,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:47,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:48,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:49,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:49,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:50,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:51,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:52,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:53,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:54,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:54,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:32:55,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:56,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:56,554 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:32:57,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:58,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:32:59,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:00,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:00,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:01,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:02,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:03,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:04,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:05,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:05,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:06,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:06,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:33:06,511 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:33:06,511 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:33:06,512 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:33:06,893 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:33:07,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:08,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:08,895 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:33:09,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:10,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:11,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:11,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:12,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:13,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:14,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:15,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:16,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:16,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:17,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:18,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:19,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:20,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:21,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:21,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:22,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:23,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:24,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:25,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:26,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:26,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:26,558 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:33:27,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:28,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:29,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:30,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:31,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:31,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:32,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:33,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:33,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:33:33,351 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:33:33,352 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:33:33,353 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:33:33,925 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:33:34,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:34,925 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:33:35,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:36,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:37,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:37,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:38,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:39,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:40,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:41,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:42,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:42,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:43,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:44,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:45,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:46,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:47,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:47,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:48,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:49,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:50,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:51,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:52,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:52,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:53,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:54,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:55,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:56,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:56,562 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:33:57,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:57,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:33:58,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:33:59,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:00,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:01,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:02,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:02,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:03,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:04,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:04,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:34:04,464 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:34:04,464 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:34:04,466 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:34:04,956 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:34:05,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:06,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:06,957 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:34:07,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:08,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:08,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:09,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:10,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:11,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:12,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:13,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:13,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:14,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:15,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:16,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:17,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:18,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:18,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:19,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:20,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:21,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:22,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:23,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:23,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:24,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:25,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:26,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:26,566 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:34:27,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:28,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:28,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:29,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:30,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:31,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:32,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:33,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:33,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:34,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:35,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:36,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:37,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:34:37,032 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:34:37,033 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:34:37,034 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:34:37,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:37,982 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:34:38,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:38,983 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:34:39,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:39,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:40,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:41,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:42,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:43,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:44,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:44,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:45,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:46,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:47,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:48,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:49,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:49,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:50,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:51,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:52,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:53,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:54,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:34:54,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:55,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:56,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:56,570 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:34:57,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:58,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:59,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:34:59,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:00,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:01,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:02,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:03,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:04,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:04,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:05,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:06,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:07,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:07,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:35:07,998 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:35:07,999 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:35:08,000 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:35:08,019 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:35:08,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:09,020 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:35:09,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:10,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:10,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:11,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:12,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:13,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:14,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:15,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:15,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:16,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:17,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:18,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:19,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:20,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:20,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:21,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:22,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:23,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:24,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:25,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:25,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:26,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:26,574 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:35:27,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:28,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:29,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:30,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:30,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:31,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:32,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:33,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:34,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:35,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:35,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:36,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:37,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:38,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:39,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:40,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:40,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:40,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:35:40,671 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:35:40,672 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:35:40,673 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:35:41,050 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:35:41,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:42,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:43,052 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:35:43,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:44,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:45,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:45,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:46,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:47,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:48,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:49,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:50,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:50,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:51,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:52,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:53,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:54,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:55,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:55,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:35:56,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:56,578 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:35:57,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:58,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:35:59,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:00,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:01,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:01,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:02,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:03,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:04,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:05,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:06,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:06,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:07,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:08,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:09,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:36:09,315 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:36:09,315 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:36:09,316 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:36:09,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:10,079 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:36:10,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:11,080 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:36:11,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:12,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:12,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:13,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:14,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:15,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:16,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:17,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:17,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:18,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:19,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:20,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:21,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:22,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:22,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:23,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:24,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:25,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:26,581 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:36:26,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:27,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:27,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:28,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:29,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:30,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:31,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:32,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:32,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:33,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:34,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:35,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:36,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:37,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:37,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:38,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:39,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:40,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:41,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:36:41,252 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:36:41,252 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:36:41,254 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:36:41,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:42,111 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:36:42,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:43,112 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:36:43,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:43,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:44,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:45,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:46,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:47,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:48,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:48,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:49,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:50,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:51,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:52,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:53,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:53,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:54,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:55,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:56,586 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:36:56,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:57,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:58,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:36:58,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:36:59,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:00,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:01,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:02,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:03,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:03,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:04,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:05,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:06,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:07,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:08,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:08,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:09,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:10,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:11,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:37:11,843 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:37:11,843 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:37:11,844 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:37:12,147 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:37:12,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:12,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:13,148 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:37:13,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:13,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:14,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:15,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:16,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:17,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:18,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:18,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:19,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:20,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:21,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:22,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:23,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:23,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:24,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:25,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:26,589 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:37:26,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:27,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:28,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:29,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:29,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:30,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:31,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:32,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:33,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:34,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:34,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:35,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:36,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:37,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:39,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:39,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:39,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:40,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:37:40,173 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:37:40,173 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:37:40,174 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:37:40,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:41,171 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:37:41,172 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:37:41,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:42,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:43,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:44,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:45,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:45,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:46,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:47,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:48,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:49,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:50,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:50,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:51,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:52,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:53,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:54,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:55,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:37:55,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:56,593 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:37:56,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:57,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:58,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:37:59,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:00,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:00,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:01,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:02,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:03,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:05,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:05,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:06,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:07,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:08,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:09,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:10,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:10,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:38:10,540 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:38:10,540 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:38:10,541 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:38:11,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:11,197 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:38:11,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:12,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:13,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:13,199 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:38:14,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:15,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:16,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:16,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:17,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:18,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:19,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:20,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:21,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:21,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:22,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:23,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:24,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:25,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:26,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:26,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:26,597 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:38:27,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:28,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:29,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:30,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:31,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:31,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:32,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:33,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:34,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:35,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:36,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:36,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:37,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:38,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:39,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:40,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:40,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:38:40,676 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:38:40,676 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:38:40,677 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:38:41,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:41,223 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:38:41,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:42,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:43,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:43,224 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:38:44,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:45,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:46,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:46,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:47,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:48,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:49,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:50,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:51,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:51,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:52,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:53,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:54,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:55,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:56,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:56,601 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:38:57,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:57,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:38:58,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:38:59,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:00,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:01,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:02,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:02,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:03,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:04,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:05,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:06,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:07,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:07,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:08,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:09,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:10,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:11,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:12,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:12,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:39:12,174 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:39:12,175 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:39:12,176 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:39:12,248 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:39:13,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:13,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:13,249 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:39:14,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:15,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:16,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:17,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:18,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:18,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:19,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:20,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:21,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:22,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:23,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:23,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:24,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:25,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:26,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:26,605 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:39:27,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:28,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:28,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:29,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:30,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:31,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:32,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:33,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:33,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:34,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:35,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:36,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:37,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:38,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:38,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:39,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:40,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:41,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:41,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:39:41,930 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:39:41,931 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:39:41,932 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:39:42,280 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:39:42,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:43,281 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:39:43,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:43,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:44,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:45,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:46,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:47,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:48,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:48,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:49,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:50,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:51,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:52,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:53,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:53,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:39:54,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:55,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:56,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:56,609 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:39:57,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:58,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:59,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:39:59,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:00,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:01,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:02,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:03,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:04,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:04,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:05,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:06,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:07,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:08,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:09,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:09,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:10,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:11,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:12,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:13,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:14,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:14,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:15,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:15,869 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:40:15,871 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:40:15,871 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:40:15,873 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:40:16,314 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:40:16,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:17,315 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:40:17,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:18,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:19,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:19,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:20,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:21,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:22,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:23,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:24,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:24,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:25,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:26,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:26,613 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:40:27,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:28,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:29,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:30,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:30,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:31,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:32,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:33,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:34,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:35,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:35,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:36,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:37,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:38,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:39,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:40,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:40,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:41,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:42,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:43,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:44,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:45,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:45,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:46,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:46,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:40:46,635 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:40:46,635 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:40:46,636 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:40:47,345 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:40:47,345 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:40:47,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:48,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:49,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:50,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:50,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:51,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:52,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:53,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:54,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:55,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:55,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:40:56,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:56,617 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:40:57,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:58,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:40:59,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:00,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:01,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:01,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:02,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:03,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:04,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:05,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:06,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:06,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:07,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:08,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:09,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:10,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:11,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:11,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:12,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:13,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:14,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:15,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:16,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:16,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:16,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:41:16,956 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:41:16,957 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:41:16,958 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:41:17,369 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:41:17,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:18,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:19,370 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:41:19,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:20,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:21,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:21,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:22,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:23,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:24,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:25,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:26,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:26,621 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:41:27,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:27,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:28,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:29,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:30,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:31,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:32,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:32,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:33,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:34,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:35,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:36,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:37,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:37,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:38,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:39,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:40,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:41,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:42,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:42,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:43,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:44,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:45,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:46,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:47,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:47,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:41:47,454 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:41:47,455 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:41:47,456 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:41:48,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:48,394 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:41:48,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:49,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:49,395 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:41:50,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:51,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:52,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:53,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:53,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:54,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:55,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:56,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:56,625 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:41:57,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:58,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:41:58,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:41:59,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:00,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:01,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:02,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:03,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:03,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:04,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:05,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:06,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:07,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:08,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:08,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:09,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:10,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:11,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:12,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:13,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:13,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:14,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:15,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:16,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:17,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:17,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:42:17,440 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:42:17,441 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:42:17,443 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:42:18,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:18,424 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:42:19,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:19,425 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:42:19,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:20,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:21,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:22,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:23,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:24,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:24,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:25,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:26,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:26,630 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:42:27,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:28,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:29,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:29,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:30,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:31,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:32,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:33,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:34,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:34,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:35,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:36,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:37,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:38,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:39,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:39,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:40,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:41,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:42,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:43,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:44,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:44,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:45,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:46,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:47,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:48,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:49,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:49,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:50,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:42:50,003 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:42:50,003 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:42:50,004 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:42:50,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:50,449 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:42:51,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:51,450 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:42:52,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:53,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:54,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:55,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:42:55,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:56,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:56,633 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:42:57,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:58,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:42:59,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:00,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:00,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:01,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:02,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:03,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:04,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:05,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:05,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:06,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:07,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:08,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:09,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:10,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:10,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:11,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:12,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:13,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:14,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:15,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:15,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:16,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:17,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:18,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:19,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:43:19,148 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:43:19,149 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:43:19,150 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:43:19,480 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:43:19,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:20,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:21,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:21,482 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:43:21,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:22,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:23,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:24,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:25,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:26,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:26,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:26,637 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:43:27,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:28,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:29,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:30,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:31,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:31,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:32,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:33,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:34,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:35,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:36,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:36,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:37,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:38,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:39,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:40,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:41,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:41,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:42,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:43,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:44,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:45,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:46,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:46,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:47,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:48,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:49,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:50,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:43:50,533 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:43:50,534 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:43:50,535 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:43:50,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:51,511 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:43:51,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:52,512 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:43:52,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:52,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:53,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:54,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:55,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:56,641 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:43:56,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:57,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:43:57,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:58,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:43:59,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:00,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:01,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:02,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:02,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:03,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:04,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:05,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:06,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:07,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:07,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:08,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:09,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:10,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:11,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:12,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:12,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:13,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:14,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:15,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:16,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:17,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:17,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:18,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:19,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:44:19,728 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:44:19,729 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:44:19,731 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:44:20,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:20,539 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:44:21,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:22,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:22,541 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:44:22,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:23,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:24,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:25,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:26,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:26,645 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:44:27,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:28,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:28,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:29,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:30,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:31,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:32,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:33,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:33,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:34,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:35,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:36,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:37,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:38,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:38,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:39,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:40,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:41,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:42,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:43,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:43,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:44,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:45,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:46,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:47,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:48,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:48,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:49,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:50,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:50,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:44:50,755 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:44:50,756 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:44:50,757 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:44:51,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:51,572 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:44:52,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:52,573 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:44:53,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:53,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:44:54,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:55,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:56,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:56,650 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:44:57,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:58,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:59,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:44:59,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:00,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:01,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:02,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:03,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:04,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:04,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:05,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:06,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:07,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:08,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:09,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:09,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:10,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:11,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:12,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:13,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:14,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:14,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:15,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:16,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:17,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:18,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:19,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:19,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:20,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:20,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:45:20,905 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:45:20,906 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:45:20,907 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:45:21,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:21,599 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:45:22,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:22,600 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:45:23,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:24,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:24,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:25,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:26,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:26,653 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:45:27,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:28,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:29,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:30,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:30,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:31,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:32,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:33,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:34,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:35,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:35,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:36,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:37,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:38,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:39,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:40,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:40,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:41,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:42,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:43,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:44,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:45,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:45,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:46,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:47,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:48,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:49,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:50,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:45:50,094 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:45:50,094 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:45:50,095 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:45:50,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:50,624 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:45:51,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:51,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:52,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:52,625 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:45:53,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:54,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:55,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:56,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:45:56,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:56,657 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:45:57,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:58,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:45:59,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:00,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:01,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:01,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:02,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:03,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:04,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:05,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:06,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:06,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:07,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:08,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:09,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:10,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:11,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:11,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:12,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:13,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:14,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:15,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:16,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:16,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:17,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:18,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:19,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:19,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:46:19,622 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:46:19,622 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:46:19,624 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:46:19,646 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:46:20,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:21,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:22,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:22,649 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:46:22,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:23,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:24,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:25,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:26,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:26,661 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:46:27,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:27,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:28,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:29,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:30,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:31,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:32,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:32,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:33,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:34,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:35,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:36,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:37,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:37,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:38,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:39,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:40,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:41,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:42,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:42,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:43,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:44,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:45,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:46,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:47,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:47,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:48,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:49,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:50,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:50,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:46:50,929 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:46:50,929 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:46:50,930 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:46:51,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:51,677 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:46:52,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:52,678 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:46:52,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:53,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:54,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:55,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:56,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:56,664 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:46:57,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:58,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:46:58,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:46:59,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:00,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:01,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:02,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:03,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:03,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:04,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:05,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:06,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:07,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:08,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:08,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:09,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:10,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:11,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:12,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:13,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:13,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:14,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:15,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:16,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:17,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:18,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:18,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:19,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:20,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:21,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:47:21,495 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:47:21,495 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:47:21,497 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:47:21,700 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:47:21,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:22,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:22,700 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:47:23,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:24,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:24,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:25,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:26,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:26,668 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:47:27,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:28,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:29,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:29,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:30,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:31,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:32,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:33,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:34,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:34,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:35,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:36,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:37,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:38,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:39,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:39,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:40,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:41,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:42,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:43,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:44,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:44,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:45,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:46,687 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:47,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:48,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:49,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:49,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:50,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:51,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:47:51,475 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:47:51,476 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:47:51,477 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:47:51,727 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:47:51,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:52,734 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:47:52,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:53,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:54,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:55,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:47:55,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:56,673 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:47:56,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:57,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:58,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:47:59,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:00,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:00,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:01,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:02,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:03,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:04,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:05,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:05,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:06,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:07,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:08,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:09,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:10,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:10,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:11,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:12,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:13,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:14,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:15,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:15,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:16,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:17,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:18,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:19,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:20,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:20,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:21,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:48:21,301 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:48:21,302 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:48:21,303 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:48:21,763 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:48:21,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:22,763 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:48:22,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:23,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:24,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:25,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:26,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:26,676 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:48:26,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:27,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:28,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:29,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:30,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:31,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:31,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:32,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:33,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:34,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:36,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:36,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:37,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:38,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:39,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:40,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:41,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:41,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:42,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:43,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:44,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:45,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:46,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:46,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:47,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:48,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:49,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:50,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:51,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:51,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:52,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:53,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:54,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:54,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:48:54,067 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:48:54,068 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:48:54,070 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:48:54,796 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:48:55,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:56,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:56,680 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:48:56,797 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:48:57,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:57,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:48:58,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:48:59,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:00,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:01,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:02,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:02,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:03,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:04,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:05,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:06,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:07,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:07,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:08,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:09,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:10,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:11,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:12,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:12,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:13,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:14,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:15,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:16,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:17,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:17,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:18,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:19,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:20,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:21,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:22,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:22,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:23,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:24,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:25,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:25,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:49:25,433 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:49:25,433 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:49:25,434 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:49:25,833 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:49:26,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:26,685 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:49:26,833 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:49:27,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:28,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:28,687 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:29,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:30,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:31,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:32,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:33,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:33,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:34,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:35,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:36,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:37,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:38,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:38,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:39,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:40,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:41,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:42,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:43,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:43,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:44,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:45,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:46,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:47,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:48,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:48,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:49,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:50,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:51,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:52,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:53,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:53,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:49:54,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:55,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:55,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:49:55,723 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:49:55,723 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:49:55,724 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:49:55,859 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:49:56,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:56,688 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:49:56,859 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:49:57,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:58,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:59,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:49:59,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:00,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:01,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:02,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:03,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:04,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:04,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:05,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:06,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:07,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:08,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:09,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:09,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:10,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:11,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:12,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:13,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:14,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:14,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:15,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:16,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:17,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:18,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:19,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:19,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:20,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:21,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:22,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:23,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:23,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:50:23,727 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:50:23,727 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:50:23,729 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:50:23,885 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:50:24,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:24,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:24,885 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:50:25,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:26,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:26,692 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:50:27,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:28,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:29,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:30,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:30,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:31,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:32,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:33,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:34,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:35,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:35,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:36,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:37,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:38,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:39,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:40,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:40,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:41,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:42,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:43,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:44,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:45,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:45,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:46,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:47,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:48,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:49,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:50,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:50,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:51,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:52,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:53,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:54,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:54,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:50:54,263 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:50:54,263 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:50:54,264 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:50:54,920 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:50:55,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:56,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:56,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:50:56,696 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:50:56,922 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:50:57,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:58,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:50:59,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:00,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:01,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:01,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:02,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:03,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:04,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:05,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:06,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:06,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:07,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:08,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:09,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:10,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:11,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:11,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:12,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:13,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:14,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:15,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:16,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:16,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:17,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:18,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:19,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:20,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:21,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:21,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:22,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:23,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:23,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:51:23,686 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:51:23,686 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:51:23,688 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:51:23,952 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:51:24,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:24,953 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:51:25,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:26,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:26,700 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:51:27,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:27,702 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:28,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:29,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:30,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:31,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:32,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:32,703 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:33,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:34,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:35,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:36,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:37,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:37,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:38,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:39,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:40,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:41,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:42,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:42,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:43,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:44,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:45,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:46,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:47,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:47,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:48,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:49,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:50,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:51,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:52,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:52,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:53,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:51:53,034 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:51:53,035 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:51:53,036 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:51:53,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:53,979 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:51:54,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:54,979 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:51:55,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:56,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:56,704 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:51:57,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:58,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:51:58,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:51:59,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:00,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:01,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:02,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:03,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:03,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:04,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:05,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:06,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:07,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:08,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:08,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:09,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:10,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:11,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:12,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:13,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:13,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:14,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:15,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:16,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:17,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:18,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:18,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:19,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:20,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:21,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:22,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:23,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:23,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:24,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:24,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:52:24,656 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:52:24,657 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:52:24,658 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:52:25,011 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:52:25,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:26,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:26,708 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:52:27,012 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:52:27,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:28,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:29,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:29,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:30,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:31,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:32,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:33,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:34,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:34,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:35,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:36,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:37,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:38,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:39,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:39,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:40,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:41,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:42,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:43,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:44,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:44,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:45,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:46,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:47,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:48,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:49,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:49,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:50,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:51,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:52,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:53,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:54,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:54,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:52:54,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:52:54,975 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:52:54,976 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:52:54,977 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:52:55,035 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:52:55,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:56,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:56,712 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:52:57,037 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:52:57,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:58,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:52:59,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:00,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:00,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:01,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:02,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:03,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:04,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:05,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:05,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:06,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:07,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:08,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:09,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:10,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:10,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:11,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:12,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:13,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:14,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:15,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:15,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:16,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:17,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:18,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:19,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:20,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:20,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:21,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:21,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:53:21,671 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:53:21,671 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:53:21,673 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:53:22,064 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:53:22,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:23,065 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:53:23,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:24,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:25,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:26,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:26,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:26,717 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:53:27,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:28,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:29,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:30,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:31,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:31,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:32,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:33,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:34,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:35,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:36,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:36,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:37,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:38,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:39,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:40,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:41,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:41,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:42,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:43,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:44,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:45,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:46,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:46,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:47,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:48,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:49,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:50,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:51,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:51,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:52,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:53,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:54,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:54,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:53:54,552 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:53:54,552 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:53:54,553 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:53:55,090 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:53:55,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:56,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:56,720 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:53:57,091 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:53:57,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:57,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:53:58,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:53:59,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:00,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:01,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:02,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:02,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:03,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:04,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:05,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:06,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:07,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:07,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:08,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:09,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:10,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:11,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:12,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:12,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:13,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:14,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:15,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:16,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:17,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:17,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:18,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:19,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:20,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:21,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:22,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:22,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:23,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:24,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:24,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:54:24,991 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:54:24,991 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:54:24,992 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:54:25,120 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:54:25,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:26,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:26,724 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:54:27,122 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:54:27,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:28,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:28,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:29,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:30,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:31,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:32,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:33,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:33,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:34,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:35,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:36,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:37,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:38,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:38,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:39,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:40,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:41,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:42,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:43,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:43,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:44,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:45,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:46,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:47,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:48,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:48,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:49,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:50,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:51,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:52,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:53,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:53,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:54,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:55,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:56,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:56,729 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:54:57,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:54:57,282 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:54:57,282 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:54:57,283 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:54:57,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:58,150 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:54:58,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:54:59,151 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:54:59,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:54:59,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:00,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:01,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:02,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:03,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:04,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:04,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:05,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:06,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:07,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:08,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:09,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:09,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:10,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:11,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:12,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:13,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:14,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:14,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:15,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:16,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:17,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:18,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:19,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:19,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:20,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:21,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:22,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:23,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:24,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:24,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:25,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:26,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:26,730 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:55:27,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:55:27,349 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:55:27,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:55:27,351 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:55:27,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:28,178 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:55:28,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:29,179 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:55:29,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:29,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:30,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:31,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:32,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:33,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:34,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:34,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:35,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:36,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:37,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:38,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:39,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:39,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:40,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:41,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:42,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:43,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:44,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:44,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:45,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:46,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:47,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:48,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:49,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:49,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:50,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:51,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:52,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:53,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:54,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:54,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:55,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:55:55,194 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:55:55,194 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:55:55,196 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:55:55,208 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:55:55,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:56,743 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:55:56,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:57,211 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:55:57,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:58,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:55:59,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:55:59,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:00,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:01,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:02,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:03,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:04,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:04,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:05,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:06,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:07,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:08,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:09,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:09,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:10,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:11,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:12,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:13,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:14,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:14,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:15,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:16,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:17,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:18,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:19,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:19,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:20,786 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:21,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:22,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:23,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:24,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:24,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:25,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:26,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:56:26,209 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:56:26,209 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:56:26,211 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:56:26,238 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:56:26,738 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:56:26,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:27,240 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:56:27,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:28,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:29,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:30,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:30,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:31,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:32,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:33,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:34,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:35,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:35,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:36,786 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:37,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:38,786 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:39,786 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:40,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:40,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:41,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:42,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:43,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:44,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:45,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:45,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:46,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:47,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:48,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:49,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:50,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:50,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:51,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:52,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:53,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:54,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:55,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:56:55,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:56,744 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:56:56,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:57,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:58,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:56:58,321 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:56:58,321 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:56:58,323 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:56:58,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:56:59,269 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:56:59,270 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:56:59,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:00,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:01,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:01,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:02,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:03,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:04,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:05,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:06,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:06,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:07,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:08,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:09,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:10,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:11,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:11,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:12,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:13,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:14,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:15,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:16,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:16,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:17,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:18,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:19,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:20,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:21,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:21,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:22,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:23,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:24,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:25,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:26,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:26,750 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:57:26,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:27,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:28,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:29,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:30,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:57:30,087 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:57:30,088 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:57:30,089 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:57:30,299 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:57:30,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:31,300 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:57:31,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:32,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:32,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:33,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:34,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:35,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:36,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:37,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:37,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:38,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:39,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:40,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:41,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:42,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:42,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:43,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:44,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:45,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:46,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:47,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:47,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:48,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:49,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:50,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:51,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:52,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:52,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:53,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:54,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:55,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:56,752 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:57:56,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:57,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:57:57,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:58,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:57:58,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:57:58,944 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:57:58,945 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:57:58,948 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:57:59,331 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:57:59,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:00,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:01,333 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:58:01,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:02,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:02,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:03,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:04,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:05,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:06,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:07,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:07,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:08,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:09,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:10,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:11,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:12,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:12,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:13,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:14,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:15,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:16,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:17,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:17,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:18,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:19,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:20,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:21,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:22,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:22,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:23,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:24,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:25,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:26,754 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:58:26,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:27,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:28,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:28,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:29,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:58:29,247 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:58:29,247 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:58:29,252 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:58:29,361 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:58:29,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:30,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:31,362 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:58:31,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:32,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:33,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:34,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:34,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:35,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:36,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:37,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:38,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:39,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:39,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:40,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:41,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:42,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:43,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:44,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:44,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:45,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:46,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:47,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:48,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:49,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:49,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:50,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:51,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:52,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:53,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:54,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:54,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:55,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:56,761 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:58:56,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:57,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:58,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:58:59,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:58:59,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:58:59,944 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:58:59,945 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:58:59,951 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:59:00,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:00,397 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:59:01,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:01,398 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:59:02,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:03,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:04,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:04,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:05,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:06,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:07,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:08,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:09,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:09,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:10,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:11,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:12,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:13,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:14,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:14,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:15,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:16,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:17,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:18,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:19,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:19,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:20,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:21,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:22,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:23,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:24,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:24,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:25,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:26,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:26,760 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:59:27,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:28,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:29,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:30,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:30,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:31,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:32,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:32,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 09:59:32,686 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 09:59:32,686 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 09:59:32,688 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 09:59:33,426 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 09:59:33,427 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 09:59:33,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:34,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:35,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:36,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:36,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:37,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:38,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:39,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:40,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:41,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:41,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:42,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:43,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:44,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:45,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:46,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:46,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:47,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:48,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:49,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:50,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:51,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:51,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:52,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:53,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:54,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:55,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:56,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:56,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 09:59:56,768 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 09:59:57,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:58,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 09:59:59,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:00,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:01,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:01,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:02,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:00:02,326 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:00:02,327 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:00:02,328 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:00:02,454 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:00:03,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:03,455 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:00:04,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:05,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:06,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:07,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:07,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:08,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:09,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:10,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:11,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:12,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:12,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:13,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:14,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:15,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:16,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:17,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:17,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:18,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:19,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:20,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:21,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:22,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:22,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:23,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:24,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:25,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:26,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:26,769 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:00:27,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:27,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:28,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:29,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:30,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:31,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:32,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:32,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:33,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:33,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:00:33,746 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:00:33,747 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:00:33,749 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:00:34,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:34,482 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:00:35,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:35,483 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:00:36,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:37,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:37,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:38,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:39,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:40,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:41,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:42,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:42,788 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:43,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:44,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:45,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:46,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:47,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:47,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:48,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:49,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:50,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:51,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:52,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:52,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:53,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:54,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:55,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:56,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:56,773 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:00:57,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:58,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:00:58,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:00:59,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:00,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:01,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:02,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:03,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:03,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:04,167 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:01:04,168 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:01:04,169 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:01:04,172 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:01:04,509 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:01:04,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:05,510 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:01:05,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:06,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:07,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:08,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:09,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:09,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:10,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:11,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:12,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:13,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:14,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:14,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:15,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:16,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:17,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:18,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:19,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:19,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:20,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:21,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:22,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:23,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:24,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:24,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:25,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:26,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:26,778 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:01:27,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:28,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:29,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:29,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:30,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:31,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:32,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:33,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:33,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:01:33,907 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:01:33,908 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:01:33,909 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:01:34,535 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:01:34,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:34,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:35,536 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:01:35,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:36,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:37,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:38,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:39,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:39,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:40,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:41,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:42,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:43,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:44,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:44,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:45,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:46,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:47,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:48,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:49,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:49,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:50,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:51,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:52,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:53,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:54,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:54,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:01:55,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:56,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:56,781 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:01:57,703 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:58,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:01:59,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:00,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:00,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:01,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:02,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:03,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:02:03,018 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:02:03,018 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:02:03,020 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:02:03,560 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:02:03,561 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:02:03,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:04,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:05,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:06,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:06,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:07,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:08,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:09,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:10,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:11,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:11,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:12,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:13,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:14,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:15,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:16,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:16,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:17,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:18,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:19,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:20,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:21,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:21,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:22,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:23,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:24,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:25,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:26,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:26,785 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:02:26,795 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:27,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:28,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:29,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:30,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:31,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:31,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:32,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:33,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:02:33,093 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:02:33,094 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:02:33,095 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:02:33,585 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:02:33,586 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:02:33,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:34,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:35,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:36,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:37,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:37,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:38,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:39,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:40,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:41,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:42,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:42,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:43,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:44,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:45,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:46,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:47,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:47,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:48,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:49,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:50,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:51,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:52,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:52,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:53,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:54,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:55,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:56,789 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:02:56,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:57,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:02:57,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:58,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:02:59,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:00,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:01,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:02,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:02,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:03,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:04,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:03:04,012 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:03:04,013 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:03:04,014 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:03:04,611 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:03:04,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:05,612 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:03:05,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:06,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:07,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:08,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:08,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:09,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:10,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:11,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:12,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:13,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:13,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:14,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:15,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:16,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:17,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:18,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:18,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:19,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:20,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:21,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:22,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:23,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:23,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:24,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:25,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:26,801 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:03:26,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:27,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:28,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:28,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:29,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:30,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:31,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:32,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:03:32,350 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:03:32,350 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:03:32,351 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:03:32,635 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:03:33,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:33,636 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:03:34,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:34,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:35,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:36,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:37,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:38,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:39,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:39,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:40,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:41,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:42,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:43,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:44,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:44,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:45,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:46,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:47,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:48,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:49,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:49,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:50,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:51,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:52,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:53,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:54,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:03:54,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:55,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:56,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:56,810 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:03:57,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:58,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:59,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:03:59,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:00,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:01,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:02,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:02,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:04:02,499 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:04:02,499 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:04:02,500 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:04:02,656 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:04:03,657 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:04:03,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:04,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:05,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:05,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:06,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:07,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:08,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:09,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:10,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:10,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:11,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:12,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:13,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:14,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:15,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:15,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:16,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:17,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:18,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:19,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:20,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:20,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:21,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:22,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:23,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:24,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:25,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:25,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:26,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:26,804 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:04:27,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:28,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:29,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:30,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:30,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:31,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:32,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:33,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:34,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:35,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:04:35,220 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:04:35,221 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:04:35,224 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:04:35,684 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:04:35,684 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:04:35,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:36,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:36,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:37,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:38,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:39,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:40,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:41,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:41,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:42,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:43,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:44,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:45,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:46,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:46,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:47,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:48,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:49,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:50,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:51,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:51,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:52,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:53,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:54,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:55,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:56,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:04:56,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:56,808 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:04:57,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:58,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:04:59,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:00,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:01,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:01,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:02,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:03,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:04,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:05,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:06,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:05:06,520 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:05:06,520 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:05:06,523 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:05:06,707 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:05:06,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:07,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:07,708 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:05:07,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:08,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:09,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:10,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:11,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:12,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:12,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:13,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:14,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:15,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:16,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:17,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:17,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:18,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:19,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:20,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:21,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:22,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:22,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:23,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:24,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:25,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:26,812 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:05:26,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:27,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:27,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:28,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:29,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:30,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:31,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:32,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:32,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:33,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:34,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:35,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:35,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:05:35,985 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:05:35,986 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:05:35,987 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:05:36,737 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:05:36,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:37,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:38,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:38,739 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:05:38,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:39,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:40,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:41,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:42,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:43,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:43,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:44,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:45,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:46,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:47,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:48,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:48,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:49,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:50,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:51,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:52,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:53,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:53,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:54,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:55,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:56,816 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:05:56,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:57,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:58,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:05:58,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:05:59,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:00,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:01,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:02,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:03,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:03,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:04,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:06:04,882 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:06:04,883 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:06:04,884 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:06:05,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:05,766 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:06:06,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:06,766 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:06:07,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:08,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:08,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:09,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:10,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:11,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:12,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:13,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:13,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:14,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:15,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:16,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:17,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:18,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:18,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:19,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:20,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:21,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:22,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:23,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:23,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:24,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:25,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:26,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:26,821 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:06:27,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:28,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:29,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:29,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:30,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:31,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:32,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:33,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:34,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:34,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:35,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:36,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:37,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:38,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:38,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:06:38,687 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:06:38,687 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:06:38,688 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:06:38,790 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:06:39,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:40,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:40,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:40,792 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:06:41,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:42,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:43,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:44,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:45,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:45,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:46,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:47,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:48,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:49,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:50,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:50,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:51,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:52,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:53,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:54,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:55,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:55,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:06:56,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:56,825 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:06:57,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:58,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:06:59,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:00,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:00,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:01,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:02,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:03,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:04,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:05,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:05,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:07:05,428 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:07:05,428 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:07:05,430 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:07:05,816 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:07:06,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:06,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:06,817 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:07:07,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:08,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:09,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:10,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:11,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:11,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:12,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:13,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:14,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:15,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:16,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:16,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:17,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:18,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:19,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:20,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:21,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:21,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:22,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:23,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:24,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:25,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:26,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:26,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:26,829 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:07:27,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:28,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:29,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:30,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:31,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:31,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:32,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:33,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:33,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:07:33,985 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:07:33,986 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:07:33,987 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:07:34,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:34,853 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:07:35,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:36,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:36,854 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:07:37,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:37,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:38,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:39,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:40,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:41,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:42,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:42,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:43,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:44,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:45,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:46,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:47,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:47,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:48,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:49,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:50,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:51,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:52,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:52,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:53,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:54,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:55,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:56,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:56,832 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:07:57,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:57,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:07:58,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:07:59,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:00,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:01,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:02,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:02,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:03,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:04,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:05,138 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:06,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:07,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:07,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:08:07,525 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:08:07,526 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:08:07,527 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:08:07,881 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:08:08,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:08,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:08,882 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:08:09,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:10,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:11,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:12,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:13,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:13,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:14,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:15,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:16,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:17,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:18,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:18,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:19,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:20,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:21,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:22,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:23,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:23,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:24,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:25,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:26,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:26,835 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:08:27,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:28,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:28,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:29,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:30,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:31,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:32,167 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:33,167 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:33,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:34,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:35,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:36,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:37,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:38,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:38,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:39,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:39,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:08:39,943 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:08:39,944 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:08:39,945 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:08:40,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:40,913 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:08:41,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:42,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:42,915 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:08:43,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:43,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:44,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:45,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:46,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:47,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:48,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:48,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:49,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:50,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:51,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:52,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:53,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:53,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:08:54,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:55,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:56,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:56,839 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:08:57,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:58,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:59,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:08:59,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:00,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:01,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:02,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:03,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:04,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:04,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:05,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:06,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:07,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:08,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:09,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:09,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:09:09,421 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:09:09,421 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:09:09,422 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:09:09,942 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:09:10,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:10,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:10,943 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:09:11,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:12,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:13,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:14,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:15,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:15,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:16,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:17,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:18,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:19,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:20,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:20,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:21,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:22,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:23,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:24,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:25,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:25,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:26,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:26,844 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:09:27,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:28,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:29,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:30,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:30,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:31,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:32,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:33,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:34,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:35,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:35,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:36,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:37,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:38,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:39,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:39,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:09:39,588 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:09:39,588 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:09:39,590 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:09:39,972 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:09:40,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:40,972 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:09:41,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:41,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:42,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:43,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:44,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:45,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:46,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:46,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:47,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:48,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:49,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:50,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:51,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:51,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:52,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:53,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:54,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:55,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:56,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:56,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:09:56,847 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:09:57,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:58,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:09:59,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:00,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:01,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:01,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:02,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:03,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:04,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:05,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:06,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:06,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:07,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:08,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:09,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:09,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:10:09,898 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:10:09,898 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:10:09,900 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:10:09,996 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:10:10,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:11,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:11,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:12,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:12,998 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:10:13,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:14,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:15,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:16,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:16,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:17,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:18,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:19,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:20,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:21,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:21,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:22,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:23,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:24,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:25,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:26,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:26,851 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:10:27,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:27,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:28,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:29,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:30,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:31,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:32,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:32,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:33,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:34,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:35,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:36,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:37,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:37,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:37,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:10:37,908 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:10:37,908 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:10:37,910 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:10:38,020 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:10:38,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:39,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:40,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:41,022 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:10:41,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:42,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:42,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:43,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:44,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:45,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:46,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:47,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:47,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:48,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:49,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:50,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:51,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:52,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:52,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:53,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:54,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:55,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:56,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:56,855 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:10:57,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:58,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:10:58,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:10:59,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:00,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:01,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:02,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:03,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:03,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:04,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:05,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:11:05,320 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:11:05,320 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:11:05,321 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:11:05,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:06,044 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:11:06,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:07,045 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:11:07,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:08,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:09,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:09,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:10,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:11,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:12,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:13,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:14,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:14,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:15,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:16,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:17,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:18,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:19,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:19,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:20,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:21,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:22,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:23,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:24,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:24,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:25,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:26,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:26,859 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:11:27,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:28,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:29,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:29,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:30,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:31,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:32,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:33,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:34,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:34,869 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:35,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:36,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:11:36,574 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:11:36,575 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:11:36,576 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:11:36,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:37,073 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:11:37,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:38,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:39,074 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:11:39,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:40,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:40,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:41,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:42,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:43,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:44,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:45,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:45,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:46,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:47,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:48,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:49,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:50,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:50,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:51,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:52,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:53,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:54,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:55,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:11:55,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:56,864 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:11:56,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:57,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:58,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:11:59,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:00,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:00,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:01,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:02,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:03,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:05,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:05,871 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:05,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:06,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:07,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:08,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:09,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:12:09,640 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:12:09,640 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:12:09,642 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:12:10,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:10,126 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:12:11,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:11,128 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:12:11,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:12,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:13,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:14,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:15,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:16,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:16,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:17,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:18,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:19,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:20,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:21,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:21,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:22,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:23,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:24,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:25,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:26,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:26,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:26,867 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:12:27,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:28,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:29,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:30,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:31,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:31,870 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:32,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:33,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:34,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:35,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:36,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:36,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:37,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:38,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:39,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:40,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:40,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:12:40,606 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:12:40,607 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:12:40,608 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:12:41,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:41,155 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:12:42,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:42,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:43,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:43,157 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:12:44,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:45,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:46,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:47,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:47,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:48,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:49,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:50,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:51,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:52,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:52,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:53,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:54,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:55,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:56,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:56,871 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:12:57,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:57,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:12:58,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:12:59,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:00,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:01,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:02,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:02,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:03,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:04,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:05,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:06,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:07,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:07,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:08,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:09,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:10,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:11,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:12,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:12,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:13:12,774 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:13:12,774 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:13:12,776 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:13:13,190 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:13:13,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:13,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:14,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:15,192 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:13:15,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:16,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:17,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:18,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:18,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:19,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:20,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:21,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:22,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:23,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:23,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:24,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:25,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:26,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:26,875 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:13:27,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:28,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:28,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:29,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:30,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:31,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:32,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:33,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:33,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:34,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:35,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:36,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:37,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:38,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:38,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:39,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:40,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:41,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:42,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:43,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:43,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:44,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:44,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:13:44,506 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:13:44,506 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:13:44,507 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:13:45,220 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:13:45,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:46,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:47,221 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:13:47,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:48,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:49,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:49,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:50,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:51,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:52,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:53,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:54,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:54,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:13:55,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:56,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:56,879 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:13:57,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:58,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:59,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:13:59,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:00,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:01,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:02,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:03,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:04,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:04,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:05,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:06,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:07,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:08,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:09,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:09,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:10,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:11,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:12,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:13,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:14,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:14,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:15,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:16,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:17,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:18,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:14:18,288 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:14:18,289 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:14:18,290 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:14:18,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:19,254 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:14:19,254 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:14:19,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:20,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:20,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:21,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:22,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:23,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:24,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:25,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:25,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:26,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:26,883 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:14:27,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:28,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:29,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:30,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:30,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:31,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:32,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:33,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:34,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:35,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:35,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:36,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:37,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:38,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:39,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:40,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:40,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:41,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:42,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:43,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:44,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:45,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:45,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:46,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:47,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:48,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:49,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:14:49,359 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:14:49,359 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:14:49,361 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:14:49,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:50,276 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:14:50,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:51,277 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:14:51,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:51,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:52,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:53,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:54,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:55,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:56,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:14:56,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:56,888 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:14:57,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:58,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:14:59,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:00,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:01,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:01,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:03,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:03,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:04,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:05,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:06,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:06,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:07,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:08,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:09,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:10,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:11,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:11,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:13,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:13,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:15,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:16,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:16,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:17,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:18,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:18,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:15:18,477 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:15:18,478 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:15:18,479 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:15:19,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:19,307 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:15:19,307 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:15:20,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:21,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:22,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:22,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:23,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:24,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:25,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:26,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:26,891 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:15:27,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:27,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:28,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:29,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:30,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:31,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:32,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:32,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:33,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:34,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:35,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:36,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:37,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:37,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:38,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:39,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:40,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:41,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:42,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:42,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:43,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:44,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:45,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:46,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:47,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:47,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:48,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:49,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:49,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:15:49,888 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:15:49,889 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:15:49,890 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:15:50,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:50,333 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:15:51,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:51,334 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:15:52,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:52,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:53,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:54,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:55,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:56,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:56,895 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:15:57,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:58,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:15:58,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:15:59,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:00,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:01,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:02,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:03,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:03,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:04,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:05,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:06,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:07,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:08,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:08,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:09,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:10,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:11,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:12,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:13,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:13,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:14,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:15,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:16,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:17,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:18,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:18,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:19,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:20,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:21,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:16:21,072 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:16:21,072 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:16:21,074 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:16:21,369 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:16:21,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:22,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:23,371 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:16:23,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:24,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:24,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:25,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:26,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:26,899 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:16:27,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:28,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:29,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:29,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:30,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:31,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:32,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:33,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:34,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:34,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:35,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:36,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:37,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:38,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:39,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:39,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:40,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:41,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:42,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:43,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:44,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:44,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:45,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:46,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:47,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:48,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:49,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:49,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:50,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:51,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:52,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:16:52,099 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:16:52,099 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:16:52,100 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:16:52,403 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:16:52,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:53,404 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:16:53,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:54,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:55,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:16:55,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:56,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:56,903 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:16:57,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:58,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:16:59,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:00,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:00,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:01,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:02,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:03,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:04,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:05,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:05,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:06,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:07,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:08,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:09,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:10,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:10,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:11,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:12,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:13,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:14,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:15,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:15,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:16,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:17,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:18,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:19,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:20,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:20,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:21,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:22,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:23,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:24,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:25,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:17:25,164 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:17:25,164 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:17:25,166 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:17:25,430 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:17:25,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:26,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:26,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:26,907 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:17:27,431 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:17:27,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:28,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:29,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:30,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:31,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:31,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:32,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:33,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:34,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:35,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:36,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:36,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:37,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:38,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:39,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:40,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:41,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:41,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:42,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:43,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:44,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:45,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:46,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:46,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:47,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:48,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:49,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:50,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:51,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:51,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:52,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:53,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:54,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:55,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:17:55,133 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:17:55,133 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:17:55,135 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:17:55,456 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:17:55,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:56,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:56,910 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:17:57,458 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:17:57,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:57,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:17:58,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:17:59,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:00,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:01,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:02,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:02,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:03,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:04,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:05,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:06,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:07,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:07,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:08,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:09,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:10,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:11,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:12,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:12,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:13,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:14,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:15,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:16,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:17,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:17,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:18,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:19,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:20,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:21,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:22,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:22,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:23,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:24,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:25,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:26,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:26,915 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:18:27,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:18:27,427 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:18:27,428 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:18:27,429 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:18:27,488 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:18:27,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:28,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:28,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:29,490 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:18:29,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:30,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:31,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:32,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:33,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:33,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:34,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:35,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:36,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:37,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:38,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:38,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:39,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:40,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:41,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:42,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:43,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:43,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:44,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:45,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:46,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:47,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:48,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:48,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:49,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:50,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:51,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:52,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:53,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:53,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:54,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:55,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:56,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:56,916 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:18:57,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:58,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:18:58,168 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:18:58,168 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:18:58,173 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:18:58,515 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:18:58,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:18:59,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:18:59,516 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:18:59,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:00,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:01,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:02,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:03,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:04,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:04,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:05,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:06,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:07,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:08,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:09,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:09,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:10,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:11,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:12,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:13,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:14,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:14,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:15,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:16,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:17,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:18,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:19,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:19,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:20,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:21,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:22,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:23,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:24,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:24,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:25,870 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:26,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:26,922 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:19:27,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:28,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:29,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:19:29,761 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:19:29,761 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:19:29,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:29,766 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:19:30,167 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:30,545 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:19:30,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:31,546 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:19:31,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:32,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:33,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:34,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:34,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:35,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:36,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:37,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:38,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:39,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:39,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:40,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:41,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:42,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:43,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:44,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:44,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:45,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:46,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:47,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:48,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:49,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:49,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:50,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:51,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:52,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:53,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:54,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:19:54,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:56,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:56,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:56,932 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:19:57,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:58,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:59,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:19:59,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:00,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:20:00,306 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:20:00,306 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:20:00,309 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:20:00,574 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:20:00,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:01,575 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:20:01,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:02,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:03,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:04,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:05,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:05,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:06,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:07,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:08,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:09,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:10,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:10,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:11,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:12,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:13,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:14,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:15,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:15,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:16,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:17,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:18,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:19,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:20,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:20,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:21,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:22,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:23,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:24,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:25,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:25,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:26,926 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:20:26,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:27,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:28,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:20:28,328 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:20:28,328 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:20:28,330 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:20:28,603 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:20:28,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:29,604 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:20:29,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:30,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:30,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:31,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:32,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:33,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:34,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:35,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:35,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:36,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:37,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:38,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:39,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:40,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:40,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:41,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:42,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:43,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:44,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:45,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:45,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:46,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:47,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:48,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:49,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:50,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:50,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:51,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:52,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:53,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:54,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:55,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:20:55,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:56,935 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:20:56,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:57,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:58,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:20:59,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:00,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:21:00,320 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:21:00,321 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:21:00,322 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:21:00,636 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:21:00,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:01,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:01,638 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:21:01,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:02,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:03,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:04,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:05,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:06,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:06,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:07,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:08,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:09,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:10,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:11,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:11,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:12,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:14,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:15,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:16,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:16,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:17,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:18,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:19,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:20,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:21,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:21,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:22,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:23,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:24,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:25,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:26,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:26,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:26,936 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:21:27,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:28,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:29,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:30,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:31,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:31,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:32,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:33,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:33,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:21:33,756 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:21:33,756 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:21:33,757 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:21:34,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:34,676 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:21:35,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:35,677 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:21:36,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:37,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:37,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:38,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:39,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:40,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:41,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:42,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:42,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:43,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:44,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:45,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:46,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:47,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:47,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:48,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:49,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:50,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:51,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:52,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:52,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:53,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:54,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:55,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:56,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:56,940 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:21:57,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:57,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:21:58,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:21:59,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:00,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:01,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:02,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:02,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:03,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:04,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:22:04,108 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:22:04,109 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:22:04,110 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:22:04,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:04,701 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:22:05,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:05,702 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:22:06,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:07,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:08,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:08,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:09,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:10,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:11,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:12,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:13,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:13,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:14,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:15,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:16,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:17,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:18,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:18,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:19,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:20,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:21,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:22,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:23,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:23,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:24,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:25,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:26,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:26,944 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:22:27,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:28,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:28,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:29,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:30,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:31,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:32,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:33,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:33,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:22:33,737 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:22:33,737 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:22:33,739 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:22:34,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:34,731 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:22:34,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:35,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:35,732 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:22:36,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:37,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:38,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:39,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:39,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:40,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:41,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:42,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:43,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:44,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:44,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:45,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:46,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:47,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:48,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:49,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:49,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:50,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:51,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:52,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:53,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:54,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:54,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:22:55,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:56,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:56,948 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:22:57,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:58,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:59,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:22:59,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:00,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:01,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:02,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:03,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:04,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:04,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:05,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:23:05,026 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:23:05,027 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:23:05,028 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:23:05,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:05,754 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:23:05,754 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:23:06,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:07,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:08,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:09,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:10,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:10,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:11,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:12,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:13,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:14,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:15,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:15,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:16,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:17,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:18,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:19,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:20,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:20,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:21,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:22,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:23,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:24,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:25,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:25,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:26,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:26,952 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:23:27,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:28,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:29,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:30,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:30,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:31,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:32,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:33,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:34,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:34,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:23:34,632 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:23:34,633 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:23:34,634 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:23:34,789 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:23:35,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:35,791 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:23:36,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:36,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:37,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:38,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:39,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:40,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:41,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:41,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:42,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:43,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:44,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:45,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:46,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:46,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:47,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:48,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:49,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:50,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:51,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:51,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:52,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:53,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:54,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:55,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:56,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:56,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:23:56,959 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:23:57,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:58,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:23:59,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:00,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:01,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:01,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:02,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:24:02,288 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:24:02,288 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:24:02,290 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:24:02,813 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:24:03,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:03,814 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:24:04,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:05,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:06,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:07,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:07,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:08,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:09,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:10,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:11,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:12,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:12,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:13,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:14,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:15,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:16,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:17,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:17,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:18,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:19,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:20,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:21,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:22,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:22,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:23,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:24,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:25,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:26,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:26,961 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:24:27,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:27,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:28,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:29,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:30,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:31,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:32,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:32,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:33,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:34,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:34,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:24:34,759 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:24:34,760 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:24:34,765 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:24:34,850 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:24:35,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:35,851 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:24:36,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:37,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:38,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:38,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:39,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:40,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:41,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:42,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:43,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:43,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:44,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:45,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:46,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:47,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:48,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:48,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:49,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:50,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:51,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:52,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:53,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:53,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:54,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:55,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:56,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:56,965 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:24:57,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:58,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:24:58,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:24:59,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:00,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:01,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:02,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:03,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:03,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:04,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:05,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:25:05,183 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:25:05,183 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:25:05,185 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:25:05,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:05,878 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:25:05,879 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:25:06,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:07,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:08,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:09,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:09,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:10,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:11,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:12,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:13,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:14,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:14,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:15,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:16,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:17,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:18,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:19,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:19,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:20,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:21,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:22,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:23,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:24,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:24,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:25,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:26,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:26,970 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:25:27,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:28,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:29,895 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:25:30,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:31,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:35,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:37,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:25:38,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:25:38,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:39,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:40,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:45,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:45,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:25:45,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:25:46,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:47,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:50,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:53,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:25:54,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:25:54,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:55,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:25:55,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:25:56,973 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:26:00,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:02,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:26:02,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:26:02,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:03,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:05,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:08,531 ERROR gpu :3286844 [interfaces.py:monitor():142] Failed to sample metric: Not Supported 2025-01-04 10:26:10,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:26:10,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:10,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:11,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:12,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:15,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:18,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:26:20,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:22,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:26:22,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:22,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:23,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:24,253 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:26:24,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:25,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:26,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:26,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:26,977 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:26:27,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:28,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:29,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:30,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:31,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:31,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:32,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:33,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:34,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:35,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:36,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:36,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:37,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:38,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:39,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:40,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:41,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:41,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:42,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:43,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:44,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:45,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:46,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:46,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:47,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:48,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:49,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:50,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:51,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:51,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:52,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:53,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:54,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:55,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:56,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:56,981 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:26:56,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:26:57,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:58,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:26:59,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:00,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:01,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:01,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:02,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:03,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:04,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:05,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:06,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:06,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:07,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:08,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:09,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:10,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:11,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:11,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:12,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:13,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:14,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:16,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:21,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:26,986 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:27:27,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:32,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:35,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:27:35,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:36,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:37,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:37,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:38,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:39,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:40,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:41,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:42,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:42,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:43,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:44,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:45,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:46,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:47,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:47,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:48,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:49,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:50,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:51,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:52,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:52,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:53,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:54,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:55,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:56,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:56,990 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:27:57,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:57,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:27:58,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:27:59,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:00,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:01,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:02,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:02,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:03,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:04,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:05,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:06,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:07,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:07,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:08,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:09,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:10,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:11,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:12,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:12,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:13,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:14,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:15,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:16,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:17,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:17,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:18,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:19,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:20,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:21,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:22,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:22,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:23,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:24,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:25,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:26,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:26,994 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:28:27,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:27,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:28,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:29,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:30,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:31,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:32,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:32,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:33,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:34,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:35,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:36,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:37,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:37,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:38,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:39,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:40,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:41,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:42,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:42,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:43,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:44,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:45,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:46,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:47,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:47,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:48,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:49,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:50,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:51,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:52,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:53,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:53,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:54,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:55,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:56,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:56,996 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:28:57,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:58,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:28:58,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:28:59,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:00,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:01,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:02,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:03,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:03,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:04,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:05,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:06,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:07,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:08,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:09,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:09,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:10,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:11,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:12,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:13,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:14,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:14,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:15,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:16,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:17,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:18,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:19,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:19,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:20,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:21,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:22,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:23,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:29:24,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:27,000 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:29:29,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:34,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:39,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:44,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:49,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:54,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:29:57,005 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:30:00,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:05,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:06,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:30:06,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 10:30:06,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:07,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:08,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:09,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:10,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:10,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:10,873 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:30:11,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:12,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:13,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:14,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:14,875 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:30:15,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:16,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:16,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:17,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:18,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:19,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:20,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:21,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:21,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:22,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:23,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:24,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:25,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:26,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:26,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:27,008 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:30:27,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:28,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:29,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:30,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:31,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:32,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:32,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:33,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:34,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:35,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:36,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:37,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:37,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:38,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:39,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:39,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:30:39,479 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:30:39,480 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:30:39,481 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:30:39,898 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:30:40,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:40,899 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:30:41,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:42,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:42,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:43,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:44,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:45,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:46,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:47,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:47,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:48,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:49,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:50,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:51,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:52,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:52,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:53,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:54,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:55,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:56,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:57,012 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:30:57,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:58,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:30:58,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:30:59,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:00,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:01,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:02,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:03,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:03,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:04,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:05,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:06,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:07,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:08,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:08,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:09,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:31:09,301 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:31:09,301 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:31:09,303 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:31:09,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:09,926 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:31:10,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:10,928 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:31:11,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:12,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:13,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:13,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:14,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:15,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:16,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:17,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:18,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:18,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:19,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:20,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:21,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:22,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:23,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:23,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:24,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:25,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:26,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:27,017 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:31:27,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:28,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:29,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:29,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:30,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:31,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:32,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:33,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:34,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:34,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:35,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:36,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:31:36,011 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:31:36,011 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:31:36,012 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:31:36,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:36,957 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:31:37,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:38,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:38,958 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:31:39,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:39,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:40,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:41,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:42,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:43,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:44,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:44,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:45,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:46,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:47,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:48,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:49,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:49,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:50,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:51,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:52,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:53,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:54,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:31:54,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:55,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:56,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:57,020 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:31:57,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:58,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:31:59,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:00,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:00,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:01,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:02,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:03,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:04,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:05,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:05,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:32:05,567 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:32:05,567 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:32:05,569 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:32:05,980 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:32:06,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:06,981 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:32:07,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:08,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:09,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:10,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:10,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:11,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:12,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:13,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:14,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:15,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:15,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:16,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:17,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:18,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:19,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:20,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:20,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:21,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:22,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:23,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:24,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:25,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:25,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:26,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:27,025 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:32:27,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:28,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:29,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:30,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:31,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:31,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:32,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:33,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:34,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:35,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:36,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:36,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:37,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:38,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:39,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:39,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:32:39,055 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:32:39,055 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:32:39,056 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:32:40,010 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:32:40,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:41,012 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:32:41,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:41,156 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:42,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:43,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:44,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:45,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:46,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:46,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:47,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:48,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:49,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:50,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:51,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:51,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:52,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:53,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:54,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:55,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:56,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:56,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:32:57,028 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:32:57,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:58,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:32:59,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:00,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:01,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:02,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:02,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:03,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:04,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:05,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:06,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:07,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:07,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:07,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:33:07,416 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:33:07,416 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:33:07,417 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:33:08,034 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:33:08,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:09,035 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:33:09,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:10,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:11,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:12,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:12,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:13,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:14,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:15,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:16,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:17,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:17,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:18,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:19,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:20,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:21,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:22,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:22,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:23,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:24,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:25,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:26,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:27,033 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:33:27,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:28,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:28,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:29,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:30,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:31,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:32,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:33,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:33,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:34,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:35,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:36,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:37,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:38,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:38,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:39,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:40,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:41,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:41,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:33:41,424 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:33:41,425 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:33:41,425 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:33:42,065 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:33:42,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:43,066 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:33:43,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:43,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:44,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:45,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:46,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:47,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:48,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:48,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:49,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:50,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:51,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:52,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:53,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:53,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:54,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:55,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:56,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:57,037 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:33:57,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:58,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:33:59,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:33:59,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:00,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:01,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:02,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:03,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:04,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:04,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:05,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:06,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:07,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:08,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:09,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:09,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:10,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:10,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:34:10,161 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:34:10,162 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:34:10,163 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:34:11,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:11,091 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:34:11,091 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:34:12,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:13,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:14,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:14,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:15,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:16,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:17,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:18,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:19,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:19,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:20,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:21,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:22,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:23,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:24,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:24,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:25,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:26,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:27,040 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:34:27,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:28,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:29,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:30,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:30,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:31,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:32,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:33,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:34,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:35,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:35,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:36,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:37,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:38,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:39,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:40,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:40,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:40,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:34:40,311 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:34:40,312 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:34:40,313 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:34:41,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:41,115 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:34:42,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:43,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:43,117 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:34:44,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:45,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:45,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:46,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:47,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:48,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:49,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:50,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:50,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:51,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:52,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:53,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:54,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:55,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:55,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:34:56,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:57,044 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:34:57,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:58,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:34:59,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:00,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:01,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:01,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:02,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:03,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:04,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:05,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:06,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:06,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:07,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:08,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:09,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:10,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:11,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:11,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:12,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:12,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:35:12,256 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:35:12,257 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:35:12,258 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:35:13,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:13,152 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:35:13,153 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:35:14,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:15,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:16,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:16,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:17,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:18,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:19,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:20,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:21,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:21,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:22,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:23,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:24,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:25,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:26,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:26,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:27,048 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:35:27,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:28,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:29,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:30,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:31,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:32,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:32,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:33,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:34,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:35,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:36,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:37,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:37,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:38,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:39,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:40,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:41,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:42,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:42,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:43,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:44,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:45,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:45,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:35:45,654 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:35:45,654 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:35:45,656 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:35:46,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:46,186 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:35:47,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:47,187 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:35:47,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:48,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:49,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:50,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:51,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:52,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:52,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:53,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:54,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:55,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:56,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:57,052 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:35:57,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:58,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:35:58,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:35:59,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:00,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:01,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:02,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:03,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:03,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:04,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:05,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:06,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:07,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:08,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:08,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:09,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:10,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:11,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:12,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:13,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:13,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:14,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:15,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:16,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:17,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:17,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:36:17,204 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:36:17,205 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:36:17,206 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:36:17,211 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:36:18,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:18,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:19,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:19,213 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:36:20,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:21,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:22,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:23,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:23,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:24,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:25,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:26,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:27,057 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:36:27,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:28,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:29,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:29,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:30,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:31,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:32,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:33,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:34,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:34,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:35,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:36,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:37,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:38,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:39,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:39,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:40,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:41,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:42,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:43,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:44,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:44,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:45,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:46,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:47,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:48,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:48,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:36:48,887 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:36:48,888 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:36:48,889 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:36:49,250 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:36:49,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:49,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:50,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:51,252 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:36:51,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:52,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:53,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:54,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:54,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:36:55,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:56,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:57,061 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:36:57,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:58,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:36:59,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:00,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:00,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:01,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:02,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:03,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:04,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:05,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:05,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:06,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:07,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:08,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:09,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:10,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:10,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:11,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:12,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:13,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:14,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:15,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:15,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:16,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:17,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:18,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:37:18,058 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:37:18,058 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:37:18,059 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:37:18,280 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:37:18,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:19,281 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:37:19,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:20,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:20,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:21,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:22,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:23,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:24,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:25,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:25,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:26,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:27,064 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:37:27,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:28,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:29,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:30,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:31,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:31,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:32,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:33,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:34,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:35,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:36,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:36,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:37,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:38,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:39,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:40,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:41,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:41,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:42,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:43,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:44,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:45,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:46,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:46,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:47,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:47,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:37:47,798 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:37:47,798 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:37:47,799 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:37:48,310 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:37:48,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:49,311 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:37:49,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:50,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:51,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:51,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:52,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:53,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:54,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:55,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:56,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:56,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:37:57,069 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:37:57,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:58,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:37:59,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:00,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:01,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:02,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:02,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:03,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:04,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:05,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:06,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:07,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:07,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:08,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:09,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:10,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:11,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:12,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:12,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:13,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:14,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:15,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:16,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:16,687 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:38:16,688 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:38:16,688 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:38:16,689 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:38:17,339 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:38:17,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:17,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:18,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:19,340 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:38:19,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:20,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:21,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:22,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:22,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:23,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:24,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:25,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:26,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:27,073 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:38:27,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:28,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:28,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:29,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:30,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:31,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:32,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:33,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:33,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:34,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:35,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:36,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:37,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:38,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:38,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:39,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:40,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:41,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:42,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:43,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:43,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:44,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:45,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:46,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:47,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:38:47,384 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:38:47,384 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:38:47,385 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:38:47,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:48,372 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:38:48,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:48,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:49,373 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:38:49,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:50,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:51,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:52,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:53,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:53,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:54,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:55,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:56,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:57,077 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:38:57,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:58,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:38:59,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:38:59,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:00,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:01,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:02,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:03,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:04,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:04,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:05,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:06,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:07,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:08,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:09,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:09,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:10,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:11,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:12,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:13,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:14,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:14,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:15,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:16,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:17,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:18,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:19,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:19,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:20,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:21,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:22,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:23,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:39:23,087 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:39:23,087 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:39:23,088 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:39:23,402 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:39:23,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:24,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:24,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:25,404 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:39:25,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:26,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:27,080 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:39:27,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:28,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:29,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:30,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:30,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:31,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:32,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:33,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:34,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:35,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:35,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:36,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:37,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:38,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:39,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:40,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:40,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:41,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:42,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:43,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:44,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:45,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:45,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:46,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:47,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:48,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:49,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:50,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:50,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:51,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:52,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:53,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:54,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:39:54,879 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:39:54,880 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:39:54,881 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:39:55,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:55,433 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:39:55,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:39:56,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:57,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:57,083 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:39:57,435 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:39:58,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:39:59,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:00,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:01,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:01,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:02,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:03,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:04,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:05,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:06,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:06,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:07,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:08,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:09,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:10,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:11,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:11,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:12,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:13,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:14,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:15,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:16,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:16,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:17,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:18,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:19,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:20,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:21,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:21,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:22,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:23,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:24,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:25,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:26,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:26,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:26,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:40:26,915 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:40:26,915 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:40:26,916 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:40:27,087 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:40:27,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:27,465 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:40:28,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:29,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:29,467 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:40:30,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:31,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:32,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:32,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:33,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:34,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:35,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:36,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:37,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:37,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:38,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:39,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:40,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:41,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:42,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:42,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:43,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:44,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:45,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:46,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:47,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:47,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:48,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:49,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:50,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:51,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:52,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:52,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:53,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:54,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:55,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:55,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:40:55,736 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:40:55,736 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:40:55,738 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:40:56,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:56,492 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:40:57,092 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:40:57,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:57,493 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:40:58,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:40:58,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:40:59,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:00,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:01,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:02,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:03,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:03,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:04,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:05,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:06,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:07,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:08,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:08,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:09,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:10,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:11,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:12,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:13,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:13,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:14,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:15,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:16,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:17,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:18,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:18,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:19,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:20,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:21,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:22,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:23,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:23,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:24,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:25,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:26,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:27,097 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:41:27,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:28,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:28,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:41:28,545 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:41:28,545 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:41:28,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:28,546 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:41:29,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:29,518 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:41:29,519 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:41:30,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:31,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:32,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:33,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:33,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:34,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:35,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:36,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:37,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:38,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:38,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:39,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:40,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:41,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:42,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:43,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:43,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:44,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:45,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:46,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:47,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:48,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:48,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:49,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:50,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:51,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:52,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:53,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:53,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:54,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:55,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:56,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:57,105 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:41:57,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:58,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:59,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:41:59,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:41:59,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:41:59,462 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:41:59,462 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:41:59,464 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:41:59,550 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:42:00,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:01,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:01,552 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:42:02,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:03,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:04,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:04,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:05,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:06,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:07,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:08,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:09,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:09,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:10,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:11,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:12,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:13,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:14,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:14,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:15,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:16,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:17,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:18,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:19,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:19,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:20,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:21,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:22,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:23,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:24,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:24,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:25,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:26,167 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:27,112 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:42:27,169 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:28,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:29,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:30,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:30,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:30,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:42:30,825 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:42:30,825 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:42:30,826 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:42:31,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:31,577 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:42:31,578 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:42:32,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:33,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:34,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:35,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:35,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:36,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:37,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:38,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:39,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:40,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:40,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:41,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:42,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:43,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:44,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:45,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:45,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:46,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:47,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:48,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:49,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:50,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:50,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:51,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:52,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:53,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:54,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:55,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:55,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:42:56,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:57,112 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:42:57,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:58,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:42:59,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:00,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:01,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:01,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:02,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:02,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:43:02,478 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:43:02,479 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:43:02,482 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:43:02,611 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:43:03,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:03,612 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:43:04,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:05,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:06,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:06,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:07,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:08,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:09,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:10,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:11,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:11,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:12,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:13,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:14,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:15,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:16,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:16,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:17,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:18,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:19,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:20,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:21,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:21,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:22,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:23,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:24,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:25,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:26,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:26,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:27,113 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:43:27,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:28,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:29,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:30,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:31,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:43:31,092 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:43:31,093 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:43:31,094 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:43:31,642 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:43:31,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:32,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:32,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:33,643 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:43:33,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:34,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:35,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:36,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:37,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:37,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:38,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:39,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:40,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:41,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:42,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:42,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:43,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:44,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:45,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:46,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:47,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:47,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:48,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:49,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:50,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:51,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:52,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:52,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:53,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:54,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:55,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:56,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:57,118 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:43:57,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:58,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:43:58,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:43:59,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:00,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:01,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:44:01,414 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:44:01,415 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:44:01,416 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:44:01,672 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:44:02,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:03,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:03,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:03,674 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:44:04,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:05,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:06,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:07,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:08,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:08,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:09,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:10,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:11,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:12,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:13,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:13,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:14,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:15,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:16,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:17,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:18,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:18,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:19,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:20,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:21,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:22,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:23,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:23,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:24,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:25,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:26,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:27,129 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:44:27,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:28,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:29,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:29,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:30,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:31,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:31,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:44:31,331 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:44:31,331 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:44:31,332 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:44:31,695 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:44:32,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:33,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:33,697 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:44:34,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:34,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:35,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:36,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:37,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:38,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:39,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:39,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:40,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:41,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:42,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:43,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:44,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:44,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:45,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:46,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:47,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:48,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:49,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:49,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:50,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:51,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:52,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:53,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:54,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:44:54,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:55,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:56,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:57,128 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:44:57,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:58,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:44:59,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:00,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:45:00,100 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:45:00,101 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:45:00,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:00,102 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:45:00,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:00,721 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:45:01,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:01,722 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:45:02,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:03,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:04,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:05,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:05,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:06,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:07,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:08,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:09,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:10,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:10,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:11,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:12,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:13,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:14,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:15,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:15,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:16,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:17,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:18,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:19,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:20,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:20,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:21,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:22,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:23,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:24,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:25,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:25,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:26,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:27,132 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:45:27,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:28,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:29,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:30,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:31,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:31,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:32,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:33,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:45:33,540 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:45:33,540 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:45:33,541 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:45:33,753 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:45:33,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:34,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:35,755 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:45:35,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:36,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:36,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:37,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:38,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:39,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:40,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:41,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:41,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:42,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:43,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:44,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:45,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:46,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:46,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:47,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:49,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:50,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:51,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:51,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:52,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:53,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:54,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:55,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:56,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:56,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:45:57,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:57,136 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:45:58,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:45:59,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:00,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:01,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:02,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:02,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:03,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:04,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:05,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:06,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:06,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:46:06,572 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:46:06,572 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:46:06,573 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:46:06,784 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:46:07,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:07,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:07,785 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:46:08,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:09,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:10,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:11,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:12,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:12,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:13,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:14,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:15,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:16,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:17,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:17,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:18,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:19,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:20,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:21,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:22,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:22,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:23,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:24,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:25,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:26,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:27,140 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:46:27,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:28,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:28,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:29,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:30,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:31,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:32,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:33,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:33,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:34,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:35,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:35,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:46:35,774 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:46:35,774 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:46:35,776 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:46:35,812 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:46:36,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:37,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:37,814 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:46:38,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:38,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:39,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:40,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:41,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:42,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:43,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:43,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:44,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:45,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:46,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:47,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:48,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:48,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:49,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:50,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:51,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:52,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:53,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:53,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:54,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:55,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:56,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:57,144 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:46:57,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:58,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:46:59,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:46:59,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:00,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:01,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:02,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:03,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:03,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:47:03,785 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:47:03,785 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:47:03,786 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:47:03,841 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:47:04,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:04,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:05,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:05,844 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:47:06,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:07,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:08,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:09,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:09,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:10,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:11,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:12,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:13,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:14,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:14,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:15,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:16,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:17,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:18,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:19,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:19,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:20,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:21,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:22,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:23,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:24,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:24,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:25,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:26,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:27,152 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:47:27,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:28,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:29,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:30,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:30,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:31,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:31,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:47:31,730 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:47:31,730 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:47:31,731 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:47:31,866 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:47:32,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:33,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:33,868 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:47:34,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:35,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:35,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:36,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:37,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:38,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:39,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:40,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:40,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:41,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:42,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:43,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:44,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:45,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:45,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:46,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:47,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:48,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:49,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:50,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:50,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:51,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:52,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:53,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:54,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:55,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:47:55,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:56,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:57,152 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:47:57,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:58,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:47:59,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:00,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:01,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:01,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:02,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:03,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:04,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:05,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:06,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:48:06,074 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:48:06,075 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:48:06,077 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:48:06,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:06,896 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:48:07,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:07,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:07,897 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:48:08,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:09,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:10,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:11,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:12,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:12,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:13,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:14,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:15,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:16,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:17,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:17,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:18,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:19,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:20,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:21,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:22,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:22,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:23,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:24,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:25,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:26,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:27,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:27,156 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:48:27,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:28,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:29,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:30,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:31,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:32,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:32,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:33,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:34,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:34,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:48:34,906 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:48:34,906 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:48:34,908 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:48:34,918 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:48:35,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:35,920 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:48:36,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:37,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:37,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:38,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:39,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:40,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:41,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:42,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:42,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:43,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:44,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:45,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:46,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:47,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:47,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:48,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:49,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:50,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:51,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:52,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:52,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:53,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:54,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:55,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:56,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:57,160 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:48:57,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:58,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:48:58,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:48:59,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:00,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:01,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:02,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:03,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:03,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:04,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:49:04,114 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:49:04,114 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:49:04,115 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:49:04,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:04,947 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:49:05,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:05,948 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:49:06,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:07,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:08,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:09,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:09,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:10,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:11,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:12,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:13,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:14,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:14,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:15,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:16,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:17,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:18,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:19,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:19,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:20,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:21,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:22,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:23,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:24,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:24,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:25,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:26,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:27,164 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:49:27,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:28,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:29,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:29,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:30,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:31,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:32,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:33,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:33,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:49:33,849 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:49:33,850 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:49:33,851 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:49:33,986 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:49:34,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:34,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:35,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:35,987 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:49:36,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:37,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:38,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:39,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:39,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:40,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:41,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:42,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:43,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:44,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:44,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:45,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:46,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:47,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:48,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:49,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:49,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:50,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:51,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:52,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:53,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:54,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:54,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:49:55,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:56,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:57,168 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:49:57,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:58,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:49:59,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:00,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:00,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:01,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:02,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:03,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:04,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:05,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:05,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:06,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:50:06,121 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:50:06,121 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:50:06,122 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:50:06,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:07,014 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:50:07,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:08,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:09,016 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:50:09,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:10,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:11,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:11,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:12,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:13,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:14,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:15,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:16,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:16,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:17,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:18,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:19,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:20,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:21,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:21,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:22,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:23,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:24,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:25,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:26,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:26,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:27,172 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:50:27,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:28,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:29,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:30,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:31,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:31,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:32,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:33,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:34,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:35,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:36,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:50:36,054 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:50:36,054 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:50:36,055 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:50:36,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:37,042 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:50:37,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:37,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:38,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:39,044 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:50:39,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:40,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:41,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:42,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:42,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:43,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:44,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:45,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:46,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:47,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:47,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:48,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:49,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:50,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:51,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:52,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:52,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:53,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:54,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:55,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:56,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:57,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:50:57,176 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:50:57,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:58,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:50:59,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:00,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:01,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:02,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:02,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:03,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:04,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:05,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:51:05,830 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:51:05,831 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:51:05,832 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:51:06,065 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:51:06,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:06,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:07,870 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:07,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:08,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:09,067 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:51:09,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:10,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:11,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:12,871 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:12,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:13,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:14,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:15,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:16,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:17,872 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:18,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:19,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:20,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:21,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:22,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:22,872 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:23,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:24,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:25,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:26,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:27,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:27,179 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:51:28,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:28,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:29,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:30,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:31,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:32,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:33,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:33,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:34,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:35,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:35,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:51:35,692 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:51:35,692 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:51:35,693 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:51:36,091 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:51:36,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:37,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:38,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:38,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:39,093 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:51:39,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:40,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:41,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:42,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:43,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:43,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:44,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:45,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:46,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:47,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:48,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:48,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:49,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:50,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:51,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:52,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:53,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:53,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:51:54,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:55,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:56,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:57,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:57,182 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:51:58,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:59,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:51:59,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:00,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:01,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:02,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:02,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:52:02,958 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:52:02,958 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:52:02,960 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:52:03,115 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:52:03,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:04,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:04,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:05,117 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:52:05,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:06,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:07,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:08,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:09,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:09,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:10,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:11,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:12,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:13,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:14,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:14,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:15,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:16,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:17,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:18,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:19,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:19,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:20,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:21,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:22,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:23,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:24,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:25,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:25,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:26,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:27,191 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:52:27,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:28,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:29,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:30,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:30,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:31,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:32,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:33,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:52:33,164 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:52:33,164 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:52:33,165 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:52:33,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:34,150 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:52:34,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:35,151 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:52:35,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:35,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:36,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:37,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:38,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:39,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:40,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:40,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:41,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:42,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:43,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:44,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:45,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:45,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:46,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:47,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:48,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:49,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:50,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:50,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:51,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:52,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:53,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:54,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:55,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:52:55,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:56,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:57,193 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:52:57,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:58,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:52:59,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:00,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:01,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:01,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:02,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:03,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:04,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:04,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:53:04,974 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:53:04,974 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:53:04,977 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:53:05,177 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:53:05,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:06,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:07,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:07,179 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:53:07,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:08,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:09,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:10,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:11,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:12,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:12,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:13,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:14,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:15,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:16,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:17,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:17,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:18,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:19,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:20,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:21,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:22,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:22,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:23,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:24,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:25,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:26,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:27,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:27,196 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:53:27,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:28,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:29,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:30,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:31,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:32,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:32,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:33,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:34,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:35,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:36,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:53:36,517 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:53:36,517 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:53:36,523 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:53:36,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:37,212 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:53:37,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:37,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:38,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:39,214 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:53:39,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:40,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:41,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:42,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:42,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:43,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:44,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:45,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:46,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:47,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:47,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:48,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:49,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:50,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:51,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:52,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:52,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:53,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:54,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:55,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:56,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:57,200 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:53:57,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:58,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:53:58,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:53:59,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:00,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:01,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:02,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:03,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:03,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:04,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:05,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:06,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:54:06,891 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:54:06,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:54:06,894 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:54:07,237 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:54:07,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:08,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:08,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:09,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:09,238 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:54:10,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:11,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:12,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:13,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:13,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:14,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:15,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:16,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:17,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:18,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:18,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:19,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:20,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:21,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:22,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:23,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:23,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:24,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:25,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:26,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:27,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:27,204 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:54:28,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:29,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:29,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:30,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:31,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:32,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:33,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:34,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:34,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:35,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:36,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:37,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:37,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:54:37,522 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:54:37,523 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:54:37,526 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:54:38,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:38,261 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:54:39,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:39,262 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:54:39,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:40,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:41,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:42,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:43,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:44,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:44,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:45,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:46,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:47,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:48,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:49,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:49,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:50,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:51,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:52,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:53,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:54,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:54,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:54:55,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:56,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:57,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:57,208 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:54:58,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:54:59,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:00,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:00,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:01,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:02,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:03,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:04,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:05,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:05,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:06,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:07,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:08,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:09,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:09,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:55:09,361 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:55:09,362 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:55:09,364 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:55:10,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:10,292 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:55:10,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:11,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:11,293 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:55:12,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:13,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:14,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:15,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:15,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:16,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:17,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:18,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:19,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:20,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:20,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:21,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:22,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:23,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:24,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:25,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:25,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:26,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:27,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:27,212 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:55:28,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:29,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:30,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:31,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:31,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:32,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:33,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:34,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:35,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:36,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:36,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:37,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:38,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:39,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:39,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:55:39,717 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:55:39,717 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:55:39,720 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:55:40,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:40,319 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:55:41,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:41,320 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:55:41,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:42,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:43,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:44,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:45,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:46,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:46,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:47,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:48,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:49,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:50,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:51,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:51,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:52,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:53,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:54,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:55,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:56,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:56,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:55:57,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:57,216 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:55:58,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:55:59,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:00,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:01,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:02,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:02,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:03,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:04,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:05,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:06,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:07,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:07,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:08,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:09,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:10,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:10,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:56:10,922 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:56:10,922 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:56:10,926 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:56:11,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:11,345 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:56:12,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:12,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:13,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:13,347 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:56:14,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:15,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:16,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:17,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:17,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:18,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:19,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:20,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:21,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:22,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:22,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:23,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:24,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:25,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:26,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:27,220 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:56:27,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:28,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:28,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:29,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:30,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:31,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:32,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:33,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:33,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:34,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:35,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:36,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:37,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:38,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:38,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:39,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:39,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:56:39,921 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:56:39,921 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:56:39,922 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:56:40,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:40,365 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:56:41,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:41,366 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:56:42,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:43,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:43,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:44,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:45,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:46,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:47,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:48,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:48,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:49,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:50,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:51,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:52,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:53,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:53,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:54,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:55,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:56,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:57,224 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:56:57,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:58,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:56:59,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:56:59,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:00,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:01,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:02,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:03,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:04,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:04,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:05,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:06,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:07,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:08,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:09,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:57:09,016 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:57:09,016 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:57:09,017 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:57:09,394 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:57:09,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:10,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:10,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:11,396 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:57:11,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:12,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:13,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:14,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:15,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:15,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:16,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:17,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:18,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:19,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:20,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:20,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:21,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:22,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:23,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:24,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:25,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:25,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:26,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:27,228 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:57:27,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:28,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:29,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:30,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:30,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:31,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:32,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:33,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:34,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:35,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:35,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:36,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:37,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:38,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:39,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:40,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:57:40,214 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:57:40,214 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:57:40,216 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:57:40,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:40,425 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:57:40,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:41,426 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:57:41,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:42,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:43,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:44,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:45,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:45,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:46,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:47,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:48,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:49,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:50,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:50,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:51,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:52,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:53,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:54,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:55,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:57:55,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:56,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:57,232 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:57:57,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:58,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:57:59,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:00,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:01,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:01,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:02,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:03,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:04,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:05,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:06,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:06,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:07,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:08,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:09,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:10,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:58:10,391 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:58:10,391 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:58:10,393 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:58:10,445 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:58:10,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:11,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:11,446 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:58:11,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:12,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:13,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:14,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:15,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:16,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:16,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:17,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:18,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:19,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:20,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:21,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:21,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:22,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:23,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:24,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:25,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:26,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:26,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:27,237 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:58:27,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:28,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:29,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:30,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:31,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:32,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:32,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:33,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:34,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:35,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:36,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:37,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:37,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:38,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:39,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:40,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:41,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:58:41,694 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:58:41,694 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:58:41,696 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:58:42,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:42,477 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:58:42,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:43,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:43,478 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:58:44,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:45,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:46,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:47,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:47,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:48,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:49,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:50,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:51,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:52,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:52,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:53,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:54,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:55,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:56,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:57,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:57,240 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:58:58,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:58:58,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:58:59,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:00,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:01,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:02,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:03,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:03,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:04,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:05,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:06,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:07,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:08,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:08,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:09,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:10,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:11,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:12,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:13,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:59:13,050 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:59:13,051 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:59:13,052 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:59:13,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:13,510 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:59:14,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:14,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:15,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:15,512 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:59:16,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:17,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:18,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:19,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:19,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:20,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:21,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:22,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:23,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:24,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:24,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:25,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:26,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:27,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:27,244 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:59:28,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:29,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:29,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:30,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:31,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:32,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:33,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:34,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:34,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:35,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:36,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:37,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:38,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:39,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:39,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:40,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:41,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:42,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:43,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:44,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:44,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:45,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:46,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:46,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 10:59:46,986 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 10:59:46,987 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 10:59:46,988 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 10:59:47,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:47,545 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 10:59:48,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:49,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:49,546 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 10:59:50,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:50,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:51,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:52,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:53,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:54,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:55,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 10:59:55,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:56,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:57,248 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 10:59:57,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:58,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 10:59:59,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:00,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:00,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:01,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:02,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:03,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:04,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:05,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:05,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:06,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:07,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:08,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:09,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:10,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:10,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:11,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:12,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:13,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:14,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:15,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:15,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:16,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:17,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:18,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:19,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:20,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:20,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:20,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:00:20,980 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:00:20,980 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:00:20,981 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:00:21,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:21,573 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:00:22,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:23,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:23,575 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:00:24,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:25,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:26,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:26,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:27,251 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:00:27,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:28,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:29,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:30,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:31,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:31,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:32,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:33,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:34,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:35,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:36,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:36,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:37,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:38,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:39,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:40,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:41,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:41,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:42,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:43,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:44,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:45,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:46,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:46,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:47,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:48,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:49,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:50,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:51,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:51,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:52,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:53,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:53,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:00:53,759 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:00:53,760 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:00:53,761 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:00:54,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:54,601 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:00:55,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:55,602 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:00:56,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:56,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:00:57,255 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:00:57,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:58,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:00:59,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:00,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:01,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:02,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:02,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:03,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:04,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:05,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:06,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:07,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:07,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:08,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:09,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:10,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:11,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:12,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:12,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:13,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:14,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:15,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:16,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:17,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:17,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:18,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:19,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:20,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:21,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:22,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:22,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:23,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:24,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:01:24,120 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:01:24,120 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:01:24,123 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:01:24,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:24,626 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:01:25,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:25,627 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:01:26,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:27,258 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:01:27,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:28,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:28,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:29,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:30,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:31,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:32,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:33,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:33,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:34,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:35,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:36,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:37,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:38,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:38,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:39,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:40,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:41,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:42,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:43,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:43,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:44,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:45,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:46,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:47,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:48,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:48,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:49,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:50,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:51,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:52,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:53,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:53,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:54,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:01:54,492 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:01:54,492 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:01:54,494 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:01:54,650 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:01:54,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:55,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:55,651 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:01:56,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:57,261 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:01:57,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:58,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:01:59,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:01:59,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:00,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:01,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:02,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:03,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:04,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:04,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:05,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:06,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:07,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:08,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:09,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:09,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:10,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:11,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:12,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:13,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:14,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:14,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:15,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:16,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:17,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:18,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:19,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:19,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:20,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:21,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:22,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:23,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:24,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:02:24,034 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:02:24,034 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:02:24,035 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:02:24,677 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:02:24,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:25,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:25,678 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:02:25,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:26,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:27,265 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:02:27,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:28,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:29,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:30,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:30,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:31,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:32,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:33,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:34,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:35,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:35,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:36,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:37,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:38,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:39,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:40,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:40,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:41,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:42,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:43,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:44,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:45,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:45,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:46,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:47,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:48,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:49,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:50,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:50,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:51,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:52,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:53,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:54,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:55,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:02:55,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:56,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:02:56,115 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:02:56,115 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:02:56,117 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:02:56,707 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:02:56,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:57,269 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:02:57,708 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:02:57,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:58,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:02:59,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:00,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:01,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:01,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:02,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:03,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:04,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:05,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:06,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:06,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:07,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:08,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:09,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:10,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:11,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:11,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:12,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:13,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:14,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:15,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:16,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:16,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:17,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:18,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:19,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:20,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:21,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:21,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:22,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:22,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:03:22,796 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:03:22,797 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:03:22,798 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:03:23,736 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:03:23,736 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:03:23,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:24,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:25,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:26,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:26,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:27,273 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:03:27,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:28,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:29,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:30,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:31,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:32,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:32,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:33,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:34,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:35,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:36,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:37,276 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:37,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:38,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:39,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:40,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:41,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:42,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:42,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:43,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:44,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:45,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:46,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:47,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:47,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:48,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:49,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:50,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:51,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:03:51,658 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:03:51,658 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:03:51,659 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:03:51,764 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:03:52,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:52,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:52,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:53,766 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:03:53,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:54,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:55,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:56,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:57,277 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:03:57,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:58,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:03:58,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:03:59,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:00,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:01,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:02,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:03,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:03,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:04,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:05,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:06,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:07,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:08,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:08,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:09,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:10,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:11,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:12,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:13,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:13,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:14,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:15,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:16,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:17,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:18,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:18,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:19,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:20,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:04:20,635 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:04:20,635 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:04:20,636 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:04:20,794 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:04:21,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:21,795 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:04:22,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:23,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:23,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:24,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:25,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:26,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:27,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:27,281 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:04:28,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:29,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:29,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:30,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:31,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:32,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:33,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:34,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:34,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:35,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:36,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:37,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:38,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:39,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:39,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:40,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:41,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:42,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:43,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:44,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:44,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:45,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:46,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:47,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:48,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:49,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:49,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:50,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:50,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:04:50,528 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:04:50,528 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:04:50,530 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:04:50,823 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:04:51,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:51,824 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:04:52,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:53,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:54,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:54,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:04:55,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:56,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:57,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:57,285 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:04:58,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:04:59,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:00,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:00,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:01,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:02,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:03,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:04,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:05,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:05,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:06,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:07,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:08,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:09,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:10,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:10,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:11,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:12,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:13,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:14,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:15,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:15,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:16,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:17,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:18,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:05:18,048 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:05:18,048 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:05:18,049 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:05:18,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:18,847 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:05:19,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:19,848 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:05:20,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:21,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:21,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:22,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:23,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:24,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:25,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:26,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:26,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:27,289 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:05:27,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:28,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:29,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:30,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:31,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:31,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:32,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:33,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:34,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:35,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:36,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:36,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:37,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:38,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:39,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:40,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:41,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:41,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:42,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:43,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:44,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:45,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:46,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:46,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:47,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:48,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:05:48,271 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:05:48,271 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:05:48,273 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:05:48,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:48,877 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:05:49,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:49,878 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:05:50,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:51,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:51,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:52,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:53,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:54,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:55,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:56,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:05:56,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:57,293 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:05:57,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:58,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:05:59,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:00,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:01,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:02,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:02,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:03,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:04,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:05,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:06,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:07,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:07,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:08,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:09,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:10,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:11,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:12,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:12,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:13,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:14,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:15,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:16,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:17,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:17,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:18,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:19,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:20,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:21,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:22,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:22,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:06:22,390 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:06:22,390 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:06:22,391 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:06:22,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:22,907 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:06:23,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:23,908 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:06:24,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:25,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:26,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:27,297 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:06:27,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:28,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:28,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:29,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:30,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:31,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:32,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:33,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:33,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:34,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:35,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:36,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:37,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:38,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:38,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:39,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:40,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:41,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:42,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:43,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:43,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:44,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:45,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:46,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:47,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:48,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:48,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:49,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:50,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:51,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:52,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:53,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:53,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:06:53,438 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:06:53,439 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:06:53,441 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:06:53,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:53,931 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:06:53,932 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:06:54,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:55,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:56,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:57,301 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:06:57,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:58,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:06:59,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:06:59,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:00,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:01,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:02,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:03,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:04,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:04,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:05,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:06,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:07,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:08,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:09,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:09,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:10,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:11,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:12,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:13,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:14,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:14,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:15,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:16,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:17,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:18,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:19,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:19,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:20,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:21,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:22,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:23,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:07:23,753 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:07:23,754 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:07:23,755 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:07:23,952 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:07:24,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:24,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:24,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:25,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:25,954 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:07:26,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:27,305 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:07:27,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:28,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:29,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:30,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:30,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:31,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:32,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:33,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:34,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:35,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:35,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:36,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:37,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:38,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:39,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:40,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:40,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:41,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:42,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:43,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:44,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:45,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:45,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:46,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:47,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:48,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:49,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:50,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:50,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:51,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:52,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:07:52,212 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:07:52,212 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:07:52,213 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:07:52,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:52,976 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:07:53,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:53,977 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:07:54,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:55,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:56,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:07:56,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:57,309 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:07:57,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:58,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:07:59,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:00,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:01,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:01,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:02,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:03,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:04,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:05,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:06,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:06,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:07,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:08,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:09,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:10,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:11,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:11,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:12,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:13,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:14,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:15,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:16,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:16,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:17,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:18,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:19,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:20,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:21,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:21,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:22,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:08:22,060 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:08:22,061 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:08:22,062 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:08:22,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:23,007 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:08:23,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:24,008 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:08:24,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:25,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:26,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:27,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:27,313 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:08:27,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:28,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:29,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:30,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:31,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:32,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:32,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:33,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:34,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:35,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:36,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:37,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:37,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:38,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:39,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:40,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:41,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:42,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:42,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:43,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:44,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:45,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:46,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:47,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:47,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:48,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:49,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:50,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:51,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:51,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:08:51,971 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:08:51,972 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:08:51,973 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:08:52,032 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:08:52,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:53,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:53,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:54,033 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:08:54,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:55,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:56,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:57,318 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:08:57,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:58,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:08:58,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:08:59,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:00,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:01,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:02,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:03,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:03,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:04,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:05,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:06,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:07,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:08,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:08,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:09,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:10,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:11,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:12,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:13,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:13,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:14,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:15,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:16,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:17,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:18,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:18,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:19,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:20,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:21,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:21,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:09:21,926 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:09:21,927 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:09:21,929 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:09:22,068 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:09:22,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:23,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:23,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:24,070 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:09:24,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:25,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:26,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:27,320 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:09:27,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:28,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:29,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:29,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:30,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:31,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:32,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:33,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:34,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:34,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:35,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:36,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:37,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:38,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:39,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:39,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:40,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:41,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:42,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:43,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:44,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:44,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:45,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:46,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:47,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:48,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:49,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:49,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:50,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:09:50,777 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:09:50,777 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:09:50,779 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:09:51,100 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:09:51,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:52,101 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:09:52,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:53,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:54,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:54,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:09:55,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:56,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:57,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:57,324 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:09:58,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:09:59,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:00,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:00,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:01,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:02,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:03,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:04,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:05,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:05,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:06,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:07,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:08,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:09,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:10,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:10,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:11,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:12,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:13,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:14,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:15,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:15,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:16,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:17,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:18,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:19,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:20,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:10:20,073 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:10:20,074 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:10:20,076 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:10:20,127 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:10:20,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:21,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:21,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:22,130 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:10:22,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:23,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:24,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:25,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:26,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:26,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:27,328 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:10:27,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:28,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:29,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:30,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:31,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:31,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:32,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:33,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:34,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:35,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:36,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:36,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:37,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:38,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:39,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:40,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:41,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:41,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:42,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:43,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:44,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:45,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:46,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:46,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:47,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:48,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:49,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:50,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:10:50,447 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:10:50,447 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:10:50,449 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:10:50,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:51,157 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:10:51,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:51,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:52,158 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:10:52,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:53,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:54,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:55,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:56,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:10:56,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:57,332 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:10:57,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:58,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:10:59,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:00,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:01,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:02,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:02,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:03,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:04,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:05,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:06,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:07,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:07,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:08,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:09,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:10,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:11,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:12,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:12,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:13,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:14,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:15,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:16,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:17,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:17,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:18,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:19,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:20,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:21,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:11:21,183 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:11:21,183 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:11:21,185 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:11:21,185 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:11:21,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:22,186 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:11:22,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:23,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:23,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:24,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:25,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:26,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:27,336 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:11:27,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:28,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:28,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:29,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:30,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:31,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:32,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:33,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:33,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:34,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:35,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:36,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:37,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:38,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:38,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:39,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:40,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:41,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:42,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:43,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:43,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:44,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:45,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:46,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:47,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:48,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:48,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:49,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:50,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:11:50,365 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:11:50,366 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:11:50,367 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:11:50,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:51,217 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:11:51,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:52,218 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:11:52,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:53,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:53,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:54,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:55,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:56,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:57,340 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:11:57,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:58,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:11:59,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:11:59,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:00,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:01,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:02,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:03,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:04,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:04,781 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:05,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:06,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:07,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:08,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:09,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:09,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:10,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:11,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:12,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:13,786 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:14,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:14,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:15,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:16,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:17,788 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:18,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:19,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:19,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:12:19,555 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:12:19,555 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:12:19,557 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:12:19,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:20,245 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:12:20,246 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:12:20,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:21,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:22,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:23,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:24,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:24,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:25,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:26,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:27,342 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:12:27,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:28,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:29,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:30,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:30,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:31,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:32,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:34,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:35,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:35,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:36,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:37,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:38,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:39,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:40,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:40,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:41,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:42,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:43,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:44,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:45,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:45,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:46,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:47,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:48,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:49,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:50,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:50,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:51,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:51,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:12:51,515 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:12:51,516 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:12:51,517 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:12:52,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:52,273 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:12:53,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:53,273 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:12:54,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:55,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:55,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:12:56,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:57,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:57,346 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:12:58,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:12:59,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:00,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:01,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:01,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:02,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:03,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:04,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:05,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:06,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:06,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:07,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:08,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:09,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:10,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:11,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:11,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:12,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:13,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:14,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:15,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:16,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:16,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:17,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:18,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:19,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:20,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:21,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:21,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:22,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:13:22,011 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:13:22,012 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:13:22,014 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:13:22,300 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:13:22,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:23,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:24,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:25,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:25,303 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:13:26,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:27,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:27,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:27,350 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:13:28,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:29,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:30,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:31,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:32,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:32,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:33,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:34,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:35,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:36,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:37,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:37,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:38,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:39,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:40,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:41,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:42,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:42,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:43,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:44,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:45,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:46,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:47,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:47,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:48,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:49,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:50,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:51,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:52,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:52,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:52,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:13:52,642 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:13:52,643 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:13:52,644 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:13:53,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:53,327 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:13:54,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:55,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:55,329 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:13:56,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:57,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:57,355 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:13:58,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:13:58,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:13:59,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:00,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:01,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:02,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:03,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:03,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:04,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:05,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:06,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:07,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:08,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:08,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:09,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:10,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:11,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:12,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:13,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:13,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:14,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:15,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:16,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:17,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:18,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:18,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:19,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:20,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:21,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:22,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:23,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:23,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:23,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:14:23,569 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:14:23,570 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:14:23,571 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:14:24,355 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:14:24,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:25,355 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:14:25,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:26,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:27,359 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:14:27,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:28,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:29,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:29,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:30,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:31,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:32,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:33,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:34,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:34,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:35,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:36,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:37,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:38,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:39,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:39,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:40,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:41,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:42,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:43,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:44,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:44,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:45,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:46,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:47,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:48,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:49,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:49,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:49,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:14:49,576 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:14:49,576 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:14:49,578 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:14:50,381 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:14:50,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:51,382 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:14:51,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:52,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:53,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:54,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:54,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:14:55,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:56,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:57,363 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:14:57,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:58,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:14:59,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:00,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:00,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:01,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:02,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:03,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:04,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:05,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:05,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:06,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:07,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:08,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:09,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:10,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:10,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:11,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:12,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:13,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:14,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:15,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:15,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:16,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:17,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:18,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:19,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:19,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:15:19,731 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:15:19,731 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:15:19,733 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:15:20,411 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:15:20,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:20,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:21,412 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:15:21,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:22,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:23,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:24,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:25,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:25,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:26,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:27,367 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:15:27,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:28,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:29,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:30,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:31,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:31,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:32,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:33,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:34,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:35,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:36,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:36,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:37,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:38,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:39,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:40,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:41,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:41,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:42,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:43,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:44,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:45,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:46,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:46,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:47,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:48,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:49,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:50,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:51,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:51,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:52,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:53,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:53,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:15:53,584 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:15:53,585 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:15:53,586 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:15:54,446 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:15:54,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:55,447 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:15:55,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:56,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:56,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:15:57,371 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:15:57,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:58,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:15:59,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:00,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:01,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:02,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:02,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:03,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:04,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:05,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:06,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:07,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:07,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:08,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:09,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:10,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:11,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:12,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:12,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:13,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:14,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:15,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:16,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:17,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:17,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:18,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:19,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:20,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:21,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:22,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:22,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:23,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:23,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:16:23,601 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:16:23,601 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:16:23,603 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:16:24,475 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:16:24,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:25,476 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:16:25,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:26,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:27,375 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:16:27,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:28,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:28,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:29,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:30,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:31,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:32,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:33,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:33,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:34,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:35,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:36,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:37,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:38,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:38,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:39,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:40,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:41,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:42,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:43,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:43,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:44,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:45,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:46,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:47,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:48,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:48,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:49,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:50,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:51,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:52,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:53,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:53,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:54,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:16:54,170 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:16:54,170 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:16:54,172 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:16:54,503 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:16:54,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:55,505 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:16:55,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:56,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:57,379 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:16:57,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:58,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:16:59,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:16:59,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:00,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:01,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:02,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:03,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:04,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:04,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:05,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:06,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:07,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:08,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:09,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:09,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:10,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:11,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:12,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:13,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:14,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:14,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:15,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:16,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:17,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:18,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:19,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:19,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:20,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:21,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:22,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:23,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:24,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:24,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:25,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:25,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:17:25,814 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:17:25,814 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:17:25,816 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:17:26,535 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:17:26,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:27,383 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:17:27,536 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:17:27,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:28,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:29,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:30,385 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:30,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:31,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:32,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:33,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:34,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:35,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:35,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:36,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:37,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:38,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:39,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:40,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:40,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:41,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:42,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:43,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:44,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:45,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:45,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:46,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:47,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:48,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:49,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:50,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:50,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:51,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:52,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:53,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:54,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:55,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:17:55,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:56,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:57,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:17:57,199 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:17:57,199 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:17:57,200 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:17:57,387 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:17:57,568 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:17:57,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:58,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:17:59,569 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:17:59,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:00,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:01,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:01,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:02,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:03,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:04,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:05,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:06,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:06,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:07,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:08,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:09,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:10,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:11,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:11,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:12,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:13,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:14,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:15,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:16,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:16,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:17,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:18,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:19,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:20,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:21,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:21,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:22,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:23,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:24,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:25,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:26,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:26,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:27,396 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:18:27,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:28,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:29,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:30,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:31,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:31,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:31,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:18:31,879 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:18:31,880 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:18:31,882 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:18:32,596 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:18:32,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:33,597 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:18:33,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:34,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:35,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:36,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:36,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:37,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:38,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:39,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:40,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:41,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:41,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:42,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:43,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:44,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:45,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:46,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:46,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:47,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:48,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:49,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:50,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:51,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:51,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:52,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:53,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:54,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:55,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:56,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:18:56,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:57,400 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:18:57,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:58,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:18:59,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:00,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:01,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:19:01,065 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:19:01,066 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:19:01,067 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:19:01,623 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:19:01,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:02,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:02,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:03,624 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:19:03,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:04,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:05,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:06,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:07,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:07,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:08,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:09,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:10,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:11,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:12,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:12,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:13,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:14,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:15,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:16,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:17,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:17,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:18,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:19,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:20,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:21,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:22,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:22,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:23,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:24,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:25,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:26,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:27,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:27,401 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:19:27,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:28,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:29,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:30,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:31,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:19:31,089 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:19:31,090 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:19:31,091 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:19:31,648 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:19:31,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:32,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:32,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:33,651 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:19:33,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:34,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:35,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:36,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:37,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:37,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:38,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:39,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:40,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:41,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:42,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:42,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:44,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:45,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:46,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:47,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:47,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:48,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:49,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:50,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:51,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:52,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:52,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:53,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:54,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:55,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:56,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:57,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:57,138 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:19:57,408 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:19:58,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:19:59,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:00,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:01,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:01,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:20:01,422 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:20:01,423 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:20:01,424 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:20:01,677 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:20:02,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:02,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:03,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:03,679 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:20:04,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:05,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:06,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:07,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:07,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:08,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:09,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:10,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:11,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:12,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:12,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:13,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:14,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:15,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:16,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:17,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:17,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:18,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:19,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:20,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:21,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:22,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:22,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:23,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:24,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:25,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:26,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:27,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:27,408 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:20:28,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:28,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:29,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:30,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:31,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:32,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:32,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:20:32,379 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:20:32,379 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:20:32,384 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:20:32,707 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:20:33,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:33,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:33,707 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:20:34,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:35,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:36,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:37,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:38,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:38,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:39,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:40,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:41,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:42,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:43,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:43,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:44,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:45,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:46,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:47,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:48,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:48,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:49,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:50,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:51,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:52,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:53,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:53,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:20:54,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:55,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:56,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:57,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:57,413 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:20:58,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:59,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:20:59,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:00,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:01,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:02,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:03,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:04,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:04,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:05,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:05,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:21:05,719 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:21:05,720 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:21:05,726 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:21:05,730 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:21:06,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:07,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:07,731 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:21:08,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:09,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:09,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:10,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:11,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:12,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:13,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:14,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:14,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:15,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:16,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:17,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:18,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:19,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:19,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:20,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:21,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:22,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:23,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:24,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:24,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:25,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:26,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:27,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:27,416 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:21:28,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:29,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:30,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:30,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:31,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:32,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:33,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:34,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:35,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:35,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:36,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:37,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:38,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:38,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:21:38,793 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:21:38,794 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:21:38,799 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:21:39,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:39,757 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:21:40,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:40,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:41,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:41,758 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:21:42,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:43,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:44,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:45,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:45,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:46,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:47,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:48,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:49,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:50,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:50,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:51,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:52,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:53,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:54,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:55,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:55,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:21:56,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:57,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:57,420 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:21:58,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:21:59,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:00,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:01,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:01,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:02,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:03,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:04,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:05,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:06,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:06,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:07,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:08,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:09,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:22:09,100 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:22:09,100 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:22:09,105 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:22:09,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:09,788 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:22:10,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:11,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:11,789 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:22:12,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:12,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:13,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:14,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:15,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:16,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:17,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:17,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:18,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:19,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:20,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:21,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:22,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:22,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:23,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:24,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:25,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:26,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:27,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:27,425 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:22:27,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:28,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:29,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:30,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:31,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:32,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:32,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:33,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:34,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:35,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:36,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:37,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:37,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:38,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:39,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:40,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:40,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:22:40,981 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:22:40,981 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:22:40,986 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:22:41,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:41,817 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:22:42,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:43,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:43,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:43,819 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:22:44,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:45,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:46,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:47,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:48,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:48,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:49,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:50,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:51,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:52,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:53,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:53,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:54,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:55,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:56,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:57,428 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:22:57,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:58,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:22:58,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:22:59,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:00,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:01,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:02,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:03,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:03,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:04,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:05,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:06,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:07,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:08,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:08,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:09,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:10,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:10,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:23:10,667 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:23:10,667 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:23:10,668 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:23:10,846 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:23:11,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:11,847 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:23:12,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:13,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:13,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:14,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:15,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:16,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:17,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:18,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:18,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:19,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:20,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:21,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:22,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:23,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:23,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:24,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:25,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:26,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:27,432 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:23:27,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:28,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:29,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:29,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:30,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:31,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:32,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:33,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:34,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:34,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:35,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:36,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:37,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:38,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:39,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:39,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:40,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:41,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:42,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:43,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:44,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:44,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:44,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:23:44,900 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:23:44,901 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:23:44,906 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:23:45,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:45,873 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:23:45,873 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:23:46,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:47,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:48,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:49,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:49,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:50,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:51,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:52,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:53,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:54,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:54,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:23:55,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:56,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:57,435 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:23:57,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:58,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:23:59,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:00,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:00,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:01,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:02,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:03,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:04,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:05,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:05,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:06,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:07,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:08,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:09,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:10,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:10,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:11,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:12,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:13,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:14,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:24:14,074 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:24:14,074 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:24:14,075 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:24:14,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:14,899 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:24:15,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:15,900 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:24:16,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:16,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:17,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:18,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:19,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:20,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:21,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:21,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:22,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:23,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:24,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:25,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:26,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:26,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:27,439 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:24:27,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:28,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:29,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:30,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:31,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:31,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:32,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:33,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:34,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:35,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:36,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:36,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:37,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:38,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:39,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:40,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:41,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:41,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:42,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:43,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:44,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:24:44,223 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:24:44,224 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:24:44,225 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:24:44,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:44,930 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:24:45,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:45,931 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:24:46,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:47,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:47,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:48,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:49,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:50,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:51,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:52,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:52,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:53,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:54,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:55,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:56,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:57,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:24:57,443 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:24:57,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:58,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:24:59,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:00,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:01,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:02,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:02,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:03,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:04,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:05,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:06,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:07,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:07,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:08,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:09,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:10,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:11,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:12,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:12,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:13,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:14,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:15,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:25:15,185 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:25:15,185 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:25:15,187 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:25:15,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:15,963 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:25:15,963 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:25:16,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:17,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:18,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:18,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:19,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:20,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:21,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:22,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:23,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:23,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:24,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:25,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:26,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:27,449 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:25:27,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:28,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:28,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:29,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:30,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:31,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:32,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:33,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:33,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:34,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:35,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:36,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:37,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:38,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:38,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:39,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:40,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:41,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:42,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:43,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:43,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:44,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:45,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:25:45,625 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:25:45,625 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:25:45,630 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:25:45,988 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:25:46,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:47,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:47,989 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:25:48,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:48,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:49,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:50,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:51,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:52,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:53,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:53,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:25:54,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:55,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:56,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:57,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:57,451 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:25:58,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:59,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:25:59,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:00,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:01,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:02,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:03,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:04,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:04,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:05,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:06,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:07,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:08,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:09,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:09,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:10,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:11,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:12,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:13,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:14,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:14,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:15,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:16,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:16,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:26:16,250 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:26:16,250 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:26:16,254 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:26:17,011 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:26:17,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:18,012 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:26:18,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:19,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:20,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:20,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:21,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:22,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:23,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:24,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:25,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:25,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:26,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:27,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:27,455 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:26:28,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:29,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:30,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:30,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:31,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:32,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:33,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:34,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:35,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:35,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:36,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:37,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:38,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:39,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:40,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:40,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:41,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:42,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:43,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:44,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:45,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:45,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:46,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:46,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:26:46,737 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:26:46,737 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:26:46,740 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:26:47,036 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:26:47,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:48,037 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:26:48,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:49,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:50,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:50,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:51,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:52,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:53,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:54,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:55,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:55,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:26:56,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:57,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:57,459 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:26:58,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:26:59,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:00,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:01,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:01,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:02,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:03,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:04,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:05,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:06,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:06,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:07,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:08,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:09,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:10,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:11,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:11,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:12,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:13,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:14,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:15,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:15,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:27:15,975 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:27:15,976 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:27:15,977 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:27:16,062 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:27:16,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:17,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:17,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:18,063 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:27:18,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:19,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:20,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:21,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:22,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:22,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:23,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:24,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:25,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:26,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:27,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:27,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:27,463 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:27:28,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:29,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:30,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:31,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:32,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:32,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:33,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:34,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:35,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:36,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:37,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:37,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:38,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:39,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:40,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:41,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:42,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:42,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:43,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:44,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:45,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:46,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:47,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:47,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:47,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:27:47,595 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:27:47,595 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:27:47,596 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:27:48,096 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:27:48,096 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:27:48,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:49,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:50,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:51,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:52,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:52,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:53,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:54,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:55,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:56,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:57,468 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:27:57,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:58,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:27:58,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:27:59,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:00,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:01,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:02,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:03,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:03,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:04,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:05,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:06,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:07,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:08,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:08,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:09,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:10,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:11,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:12,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:13,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:13,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:14,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:15,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:16,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:17,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:18,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:28:18,031 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:28:18,032 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:28:18,037 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:28:18,119 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:28:18,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:19,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:19,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:20,121 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:28:20,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:21,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:22,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:23,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:24,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:24,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:25,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:26,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:27,471 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:28:27,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:28,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:29,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:29,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:30,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:31,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:32,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:33,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:34,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:34,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:35,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:36,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:37,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:38,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:39,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:39,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:40,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:41,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:42,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:43,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:44,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:44,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:45,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:46,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:47,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:48,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:49,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:28:49,376 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:28:49,377 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:28:49,380 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:28:49,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:50,150 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:28:50,150 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:28:50,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:50,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:51,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:52,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:53,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:54,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:55,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:28:55,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:56,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:57,474 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:28:57,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:58,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:28:59,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:00,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:00,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:02,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:03,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:04,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:05,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:05,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:06,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:07,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:08,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:09,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:10,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:10,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:11,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:12,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:13,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:14,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:15,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:15,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:16,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:17,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:18,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:19,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:19,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:29:19,479 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:29:19,479 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:29:19,484 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:29:20,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:20,178 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:29:20,178 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:29:20,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:21,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:22,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:23,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:24,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:25,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:25,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:26,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:27,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:27,478 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:29:28,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:29,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:30,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:31,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:31,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:32,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:33,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:34,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:35,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:36,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:36,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:37,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:38,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:39,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:40,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:41,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:41,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:42,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:43,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:44,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:45,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:46,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:46,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:47,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:48,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:49,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:50,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:29:50,011 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:29:50,011 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:29:50,016 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:29:50,201 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:29:50,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:51,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:52,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:52,203 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:29:52,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:53,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:54,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:55,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:56,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:57,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:29:57,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:57,482 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:29:58,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:29:59,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:00,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:01,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:02,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:02,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:03,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:04,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:05,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:06,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:07,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:07,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:08,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:09,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:10,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:11,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:12,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:12,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:13,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:14,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:15,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:16,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:17,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:17,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:18,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:19,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:20,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:21,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:22,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:22,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:23,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:23,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:30:23,530 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:30:23,530 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:30:23,536 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:30:24,233 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:30:24,233 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:30:24,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:25,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:26,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:27,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:27,486 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:30:28,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:28,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:29,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:30,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:31,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:32,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:33,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:33,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:34,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:35,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:36,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:37,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:38,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:38,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:39,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:40,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:41,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:42,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:43,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:43,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:44,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:45,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:46,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:47,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:48,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:48,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:49,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:50,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:51,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:52,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:53,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:53,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:30:54,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:54,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:30:54,692 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:30:54,693 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:30:54,696 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:30:55,263 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:30:55,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:56,264 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:30:56,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:57,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:57,490 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:30:58,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:59,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:30:59,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:00,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:01,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:02,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:03,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:04,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:04,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:05,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:06,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:07,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:08,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:09,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:09,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:10,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:11,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:12,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:13,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:14,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:14,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:15,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:16,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:17,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:18,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:19,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:19,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:20,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:21,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:22,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:22,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:31:22,761 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:31:22,761 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:31:22,763 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:31:23,287 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:31:23,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:24,288 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:31:24,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:24,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:25,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:26,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:27,494 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:31:27,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:28,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:29,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:30,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:30,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:31,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:32,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:33,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:34,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:35,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:35,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:36,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:37,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:38,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:39,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:40,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:40,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:41,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:42,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:43,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:44,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:45,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:45,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:46,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:47,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:48,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:49,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:50,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:50,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:51,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:51,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:31:51,793 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:31:51,794 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:31:51,795 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:31:52,325 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:31:52,325 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:31:52,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:53,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:54,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:55,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:55,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:31:56,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:57,498 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:31:57,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:58,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:31:59,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:00,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:01,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:01,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:02,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:03,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:04,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:05,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:06,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:06,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:07,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:08,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:09,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:10,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:11,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:11,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:12,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:13,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:14,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:15,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:16,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:16,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:17,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:18,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:19,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:20,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:21,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:21,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:22,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:23,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:23,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:32:23,722 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:32:23,723 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:32:23,724 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:32:24,356 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:32:24,357 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:32:24,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:25,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:26,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:26,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:27,502 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:32:27,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:28,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:29,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:30,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:31,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:32,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:32,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:33,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:34,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:35,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:36,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:37,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:37,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:38,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:39,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:40,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:41,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:42,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:42,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:43,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:44,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:45,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:46,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:47,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:47,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:48,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:49,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:32:49,230 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:32:49,231 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:32:49,232 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:32:49,387 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:32:49,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:50,389 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:32:50,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:51,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:52,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:53,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:53,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:54,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:55,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:56,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:57,506 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:32:57,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:58,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:32:58,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:32:59,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:00,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:01,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:02,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:03,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:03,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:04,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:05,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:06,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:07,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:08,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:08,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:09,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:10,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:11,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:12,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:13,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:13,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:14,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:15,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:16,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:17,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:17,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:33:17,997 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:33:17,998 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:33:17,999 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:33:18,426 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:33:18,426 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:33:18,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:19,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:19,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:20,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:21,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:22,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:23,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:24,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:24,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:25,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:26,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:27,511 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:33:27,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:28,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:29,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:29,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:30,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:31,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:32,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:33,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:34,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:34,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:35,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:36,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:37,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:38,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:39,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:39,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:40,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:41,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:42,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:43,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:44,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:44,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:45,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:46,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:47,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:48,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:33:48,189 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:33:48,189 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:33:48,190 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:33:48,463 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:33:48,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:49,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:50,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:50,465 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:33:50,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:51,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:52,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:53,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:54,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:55,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:33:55,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:56,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:57,514 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:33:57,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:58,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:33:59,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:00,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:00,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:01,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:02,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:03,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:04,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:05,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:05,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:06,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:07,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:08,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:09,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:10,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:10,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:11,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:12,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:13,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:14,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:15,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:15,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:16,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:17,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:34:17,696 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:34:17,696 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:34:17,698 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:34:18,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:18,489 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:34:18,490 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:34:18,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:19,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:20,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:20,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:21,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:22,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:23,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:24,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:25,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:25,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:26,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:27,517 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:34:27,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:28,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:29,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:30,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:31,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:31,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:32,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:33,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:34,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:35,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:36,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:36,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:37,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:38,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:39,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:40,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:41,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:41,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:42,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:43,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:44,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:45,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:46,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:46,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:47,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:48,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:49,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:50,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:34:50,124 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:34:50,124 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:34:50,125 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:34:50,529 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:34:50,530 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:34:50,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:51,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:52,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:52,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:53,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:54,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:55,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:56,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:57,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:34:57,521 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:34:57,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:58,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:34:59,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:00,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:01,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:02,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:02,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:03,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:04,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:05,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:06,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:07,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:07,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:08,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:09,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:10,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:11,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:12,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:12,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:13,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:14,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:15,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:16,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:17,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:17,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:18,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:19,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:20,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:35:20,031 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:35:20,032 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:35:20,032 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:35:20,557 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:35:20,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:21,558 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:35:21,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:22,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:23,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:23,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:24,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:25,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:26,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:27,525 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:35:27,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:28,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:28,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:29,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:30,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:31,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:32,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:33,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:33,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:34,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:35,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:36,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:37,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:38,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:38,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:39,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:40,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:41,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:42,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:43,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:43,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:44,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:45,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:46,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:47,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:48,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:48,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:49,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:35:49,154 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:35:49,154 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:35:49,155 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:35:49,578 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:35:49,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:50,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:51,583 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:35:51,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:52,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:53,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:54,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:54,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:55,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:56,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:57,529 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:35:57,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:58,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:35:59,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:35:59,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:00,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:01,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:02,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:03,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:04,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:04,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:05,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:06,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:07,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:08,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:09,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:09,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:10,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:11,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:12,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:13,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:14,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:14,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:15,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:16,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:17,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:18,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:19,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:19,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:20,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:21,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:22,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:36:22,173 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:36:22,174 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:36:22,175 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:36:22,612 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:36:22,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:23,613 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:36:23,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:24,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:25,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:25,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:26,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:27,533 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:36:27,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:28,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:29,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:30,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:30,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:31,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:32,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:33,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:34,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:35,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:35,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:36,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:37,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:38,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:39,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:40,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:40,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:41,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:42,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:43,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:44,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:45,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:45,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:46,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:47,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:48,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:49,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:50,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:50,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:51,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:52,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:53,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:54,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:55,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:36:55,397 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:36:55,397 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:36:55,398 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:36:55,637 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:36:55,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:56,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:36:56,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:57,537 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:36:57,639 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:36:57,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:58,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:36:59,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:00,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:01,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:01,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:02,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:03,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:04,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:05,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:06,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:06,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:07,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:08,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:09,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:10,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:11,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:11,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:12,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:13,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:14,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:15,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:16,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:16,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:17,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:18,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:19,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:20,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:21,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:21,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:22,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:23,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:24,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:25,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:37:25,116 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:37:25,116 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:37:25,118 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:37:25,670 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:37:25,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:26,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:27,156 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:27,541 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:37:27,671 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:37:27,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:28,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:29,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:30,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:31,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:32,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:32,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:33,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:34,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:35,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:36,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:37,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:37,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:38,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:39,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:40,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:41,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:42,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:42,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:43,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:44,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:45,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:46,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:47,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:47,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:48,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:49,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:50,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:51,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:52,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:52,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:53,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:54,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:55,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:37:55,007 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:37:55,008 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:37:55,009 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:37:55,696 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:37:55,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:56,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:57,545 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:37:57,697 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:37:57,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:58,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:37:58,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:37:59,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:00,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:01,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:02,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:03,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:03,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:04,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:05,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:06,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:07,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:08,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:08,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:09,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:10,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:11,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:12,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:13,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:13,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:14,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:15,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:16,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:17,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:18,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:18,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:19,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:20,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:21,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:22,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:23,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:23,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:24,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:25,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:38:25,867 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:38:25,868 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:38:25,869 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:38:26,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:26,723 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:38:27,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:27,549 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:38:27,724 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:38:28,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:29,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:29,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:30,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:31,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:32,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:33,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:34,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:34,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:35,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:36,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:37,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:38,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:39,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:39,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:40,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:41,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:42,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:43,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:44,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:44,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:45,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:46,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:47,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:48,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:49,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:49,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:50,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:51,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:52,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:53,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:53,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:38:53,310 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:38:53,310 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:38:53,312 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:38:53,748 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:38:54,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:55,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:55,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:38:55,749 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:38:56,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:57,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:57,554 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:38:58,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:38:59,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:00,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:00,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:01,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:02,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:03,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:04,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:05,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:05,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:06,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:07,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:08,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:09,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:10,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:10,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:11,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:12,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:13,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:14,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:15,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:15,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:16,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:17,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:18,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:19,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:20,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:20,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:21,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:22,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:23,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:23,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:39:23,702 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:39:23,702 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:39:23,704 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:39:23,781 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:39:24,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:25,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:25,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:25,783 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:39:26,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:27,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:27,557 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:39:28,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:29,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:30,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:31,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:31,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:32,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:33,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:34,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:35,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:36,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:36,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:37,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:38,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:39,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:40,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:41,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:41,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:42,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:43,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:44,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:45,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:46,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:46,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:47,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:48,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:49,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:50,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:51,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:51,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:52,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:53,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:54,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:55,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:56,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:39:56,051 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:39:56,052 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:39:56,053 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:39:56,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:56,807 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:39:57,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:39:57,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:57,561 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:39:57,808 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:39:58,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:39:59,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:00,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:01,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:02,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:02,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:03,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:04,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:05,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:06,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:07,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:07,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:08,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:09,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:10,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:11,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:12,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:12,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:13,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:14,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:15,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:16,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:17,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:17,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:18,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:19,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:20,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:21,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:22,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:22,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:23,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:24,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:25,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:26,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:26,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:40:26,945 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:40:26,946 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:40:26,947 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:40:27,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:27,565 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:40:27,842 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:40:28,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:28,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:29,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:29,844 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:40:30,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:31,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:32,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:33,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:33,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:34,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:35,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:36,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:37,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:38,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:38,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:39,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:40,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:41,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:42,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:43,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:43,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:44,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:45,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:46,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:47,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:48,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:48,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:49,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:50,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:51,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:52,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:53,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:53,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:54,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:55,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:56,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:56,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:40:56,908 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:40:56,908 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:40:56,909 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:40:57,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:57,569 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:40:57,867 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:40:58,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:59,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:40:59,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:40:59,868 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:41:00,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:01,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:02,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:03,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:04,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:04,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:05,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:06,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:07,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:08,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:09,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:09,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:10,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:11,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:12,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:13,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:14,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:14,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:15,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:16,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:17,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:18,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:19,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:19,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:20,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:21,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:22,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:23,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:24,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:24,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:25,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:26,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:27,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:27,582 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:41:28,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:29,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:29,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:30,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:31,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:41:31,183 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:41:31,184 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:41:31,185 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:41:31,893 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:41:32,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:33,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:33,895 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:41:34,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:35,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:35,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:36,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:37,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:38,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:39,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:40,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:40,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:41,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:42,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:43,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:44,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:45,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:45,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:46,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:47,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:48,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:49,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:50,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:50,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:51,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:52,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:53,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:54,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:55,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:55,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:41:56,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:57,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:57,577 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:41:58,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:41:59,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:00,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:00,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:01,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:02,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:03,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:04,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:04,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:42:04,156 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:42:04,157 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:42:04,159 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:42:04,923 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:42:05,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:05,924 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:42:06,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:06,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:07,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:08,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:09,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:10,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:11,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:11,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:12,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:13,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:14,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:15,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:16,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:16,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:17,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:18,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:19,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:20,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:21,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:21,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:22,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:23,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:24,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:25,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:26,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:26,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:27,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:27,581 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:42:28,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:29,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:30,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:31,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:31,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:32,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:33,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:34,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:34,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:42:34,108 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:42:34,109 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:42:34,110 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:42:34,951 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:42:35,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:35,953 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:42:36,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:37,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:37,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:38,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:39,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:40,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:41,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:42,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:42,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:43,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:44,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:45,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:46,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:47,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:47,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:48,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:49,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:50,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:51,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:52,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:52,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:53,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:54,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:55,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:56,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:57,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:57,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:42:57,585 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:42:58,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:42:59,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:00,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:01,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:02,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:02,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:03,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:04,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:04,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:43:04,832 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:43:04,832 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:43:04,834 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:43:04,985 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:43:05,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:05,986 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:43:06,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:07,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:07,872 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:08,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:09,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:10,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:11,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:12,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:12,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:13,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:14,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:15,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:16,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:17,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:17,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:18,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:19,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:20,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:21,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:22,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:22,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:23,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:24,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:25,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:26,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:27,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:27,589 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:43:28,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:28,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:29,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:30,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:31,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:32,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:33,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:33,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:34,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:34,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:43:34,583 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:43:34,583 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:43:34,584 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:43:35,014 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:43:35,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:36,014 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:43:36,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:37,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:38,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:38,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:39,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:40,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:41,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:42,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:43,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:43,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:44,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:45,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:46,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:47,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:48,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:48,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:49,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:50,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:51,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:52,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:53,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:53,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:43:54,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:55,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:56,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:57,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:57,593 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:43:58,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:59,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:43:59,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:00,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:01,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:02,248 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:03,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:04,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:04,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:05,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:06,248 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:07,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:07,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:44:07,550 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:44:07,550 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:44:07,556 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:44:08,044 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:44:08,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:09,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:10,046 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:44:10,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:10,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:11,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:12,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:13,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:14,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:15,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:15,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:16,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:17,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:18,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:19,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:20,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:20,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:21,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:22,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:23,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:24,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:25,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:25,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:26,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:27,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:27,595 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:44:28,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:29,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:30,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:30,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:31,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:32,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:33,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:34,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:35,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:35,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:36,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:37,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:38,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:39,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:39,872 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:44:39,873 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:44:39,874 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:44:39,877 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:44:40,083 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:44:40,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:40,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:41,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:42,084 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:44:42,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:43,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:44,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:45,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:45,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:46,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:47,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:48,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:49,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:50,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:50,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:51,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:52,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:53,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:54,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:55,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:55,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:44:56,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:57,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:57,599 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:44:58,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:44:59,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:00,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:01,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:01,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:02,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:03,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:04,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:05,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:06,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:06,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:07,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:08,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:09,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:10,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:11,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:11,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:12,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:13,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:13,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:45:13,323 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:45:13,324 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:45:13,328 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:45:14,117 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:45:14,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:15,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:16,119 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:45:16,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:17,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:17,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:18,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:19,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:20,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:21,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:22,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:22,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:23,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:24,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:25,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:26,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:27,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:27,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:27,603 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:45:28,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:29,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:30,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:31,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:32,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:32,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:33,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:34,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:35,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:36,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:37,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:37,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:38,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:39,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:40,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:41,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:42,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:42,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:43,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:44,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:45,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:45,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:45:45,639 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:45:45,639 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:45:45,641 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:45:46,145 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:45:46,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:47,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:47,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:48,147 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:45:48,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:49,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:50,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:51,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:52,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:52,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:53,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:54,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:55,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:56,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:57,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:57,606 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:45:58,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:45:58,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:45:59,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:00,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:01,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:02,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:03,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:03,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:04,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:05,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:06,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:07,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:08,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:08,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:09,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:10,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:11,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:12,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:13,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:13,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:14,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:15,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:16,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:17,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:18,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:46:18,036 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:46:18,037 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:46:18,039 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:46:18,177 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:46:18,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:19,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:19,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:20,178 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:46:20,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:21,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:22,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:23,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:24,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:24,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:25,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:26,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:27,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:27,610 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:46:28,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:29,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:29,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:30,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:31,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:32,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:33,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:34,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:34,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:35,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:36,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:37,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:38,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:39,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:39,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:40,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:41,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:42,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:43,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:44,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:44,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:45,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:46,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:47,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:48,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:49,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:49,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:49,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:46:49,777 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:46:49,778 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:46:49,783 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:46:50,204 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:46:50,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:51,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:52,206 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:46:52,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:53,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:54,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:54,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:46:55,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:56,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:57,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:57,615 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:46:58,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:46:59,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:00,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:00,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:01,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:02,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:03,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:04,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:05,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:05,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:06,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:07,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:08,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:09,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:10,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:10,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:11,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:12,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:13,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:14,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:15,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:15,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:16,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:17,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:18,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:19,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:20,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:20,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:20,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:47:20,984 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:47:20,985 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:47:20,990 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:47:21,234 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:47:21,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:22,236 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:47:22,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:23,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:24,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:25,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:26,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:26,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:27,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:27,618 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:47:28,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:29,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:30,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:31,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:31,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:32,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:33,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:34,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:35,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:36,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:36,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:37,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:38,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:39,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:40,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:41,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:41,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:42,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:43,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:44,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:45,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:46,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:46,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:47,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:48,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:49,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:50,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:47:50,356 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:47:50,357 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:47:50,357 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:47:50,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:51,266 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:47:51,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:52,266 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:47:52,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:52,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:53,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:54,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:55,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:56,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:57,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:47:57,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:57,622 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:47:58,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:47:59,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:00,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:01,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:02,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:02,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:03,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:04,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:05,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:06,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:07,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:07,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:08,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:09,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:10,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:11,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:12,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:12,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:13,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:14,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:15,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:16,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:17,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:17,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:18,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:19,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:20,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:21,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:21,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:48:21,531 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:48:21,531 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:48:21,532 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:48:22,292 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:48:22,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:23,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:23,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:24,294 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:48:24,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:25,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:26,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:27,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:27,626 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:48:28,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:28,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:29,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:30,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:31,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:32,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:33,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:33,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:34,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:35,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:36,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:37,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:38,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:38,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:39,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:40,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:41,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:42,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:43,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:43,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:44,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:45,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:46,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:47,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:48,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:48,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:48,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:48:48,852 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:48:48,853 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:48:48,854 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:48:49,320 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:48:49,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:50,322 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:48:50,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:51,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:52,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:53,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:53,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:48:54,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:55,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:56,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:57,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:57,630 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:48:58,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:59,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:48:59,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:00,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:01,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:02,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:03,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:04,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:04,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:05,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:06,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:07,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:08,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:09,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:09,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:10,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:11,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:12,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:13,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:14,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:14,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:15,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:16,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:17,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:18,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:18,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:49:18,651 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:49:18,652 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:49:18,653 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:49:19,353 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:49:19,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:19,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:20,355 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:49:20,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:21,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:22,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:23,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:24,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:24,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:25,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:26,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:27,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:27,635 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:49:28,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:29,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:30,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:30,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:31,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:32,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:33,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:34,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:35,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:35,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:36,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:37,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:38,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:39,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:40,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:40,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:41,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:42,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:43,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:44,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:45,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:45,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:46,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:47,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:48,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:49,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:49:49,348 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:49:49,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:49:49,350 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:49:49,380 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:49:49,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:50,381 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:49:50,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:51,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:51,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:52,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:53,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:54,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:55,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:56,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:49:56,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:57,638 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:49:57,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:58,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:49:59,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:00,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:01,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:01,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:02,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:03,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:04,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:05,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:06,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:06,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:07,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:08,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:09,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:10,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:11,642 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:11,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:12,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:13,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:14,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:15,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:16,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:16,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:17,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:18,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:19,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:20,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:20,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:50:20,843 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:50:20,844 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:50:20,845 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:50:21,406 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:50:21,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:21,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:22,407 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:50:22,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:23,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:24,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:25,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:26,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:26,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:27,642 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:50:27,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:28,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:29,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:30,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:31,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:32,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:32,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:33,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:34,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:35,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:36,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:37,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:37,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:38,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:39,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:40,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:41,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:42,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:42,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:43,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:44,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:45,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:46,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:47,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:47,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:48,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:49,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:50,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:51,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:50:51,144 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:50:51,144 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:50:51,146 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:50:51,430 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:50:51,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:52,431 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:50:52,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:53,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:53,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:54,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:55,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:56,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:57,646 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:50:57,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:58,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:50:58,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:50:59,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:00,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:01,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:02,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:03,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:03,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:04,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:05,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:06,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:07,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:08,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:08,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:09,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:10,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:11,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:12,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:13,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:13,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:14,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:15,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:16,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:17,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:18,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:18,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:19,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:20,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:21,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:21,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:51:21,990 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:51:21,990 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:51:21,991 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:51:22,459 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:51:22,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:23,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:24,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:24,461 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:51:24,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:25,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:26,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:27,650 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:51:27,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:28,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:29,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:29,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:30,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:31,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:32,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:33,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:34,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:34,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:35,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:36,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:37,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:38,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:39,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:39,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:40,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:41,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:42,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:43,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:44,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:44,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:45,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:46,784 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:47,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:48,788 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:49,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:49,788 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:50,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:51,795 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:52,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:53,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:51:53,080 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:51:53,080 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:51:53,081 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:51:53,490 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:51:53,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:54,491 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:51:54,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:55,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:51:55,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:56,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:57,654 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:51:57,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:58,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:51:59,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:00,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:00,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:01,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:02,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:03,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:04,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:05,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:05,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:06,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:07,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:08,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:09,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:10,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:10,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:11,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:12,795 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:13,795 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:14,795 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:15,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:15,795 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:16,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:17,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:18,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:19,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:20,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:20,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:21,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:22,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:52:22,150 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:52:22,150 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:52:22,152 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:52:22,513 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:52:22,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:23,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:24,515 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:52:24,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:25,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:26,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:26,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:27,659 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:52:27,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:28,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:29,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:30,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:31,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:31,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:32,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:33,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:34,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:35,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:36,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:36,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:37,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:38,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:39,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:40,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:41,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:41,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:42,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:43,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:44,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:45,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:46,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:46,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:47,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:48,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:49,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:50,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:51,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:51,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:52,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:52,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:52:52,843 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:52:52,843 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:52:52,845 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:52:53,545 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:52:53,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:54,546 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:52:54,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:55,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:56,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:56,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:52:57,662 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:52:57,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:58,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:52:59,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:00,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:01,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:02,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:02,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:03,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:04,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:05,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:06,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:07,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:07,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:08,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:09,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:10,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:11,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:12,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:12,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:13,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:14,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:15,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:16,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:17,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:17,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:18,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:19,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:20,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:21,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:53:21,282 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:53:21,282 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:53:21,283 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:53:21,574 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:53:21,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:22,575 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:53:22,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:23,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:23,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:24,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:25,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:26,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:27,666 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:53:27,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:28,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:28,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:29,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:30,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:31,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:32,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:33,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:33,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:34,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:35,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:36,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:37,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:38,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:38,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:39,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:40,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:41,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:42,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:43,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:43,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:44,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:45,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:46,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:47,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:48,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:48,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:49,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:50,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:51,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:52,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:53:52,308 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:53:52,308 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:53:52,309 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:53:52,606 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:53:52,607 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:53:52,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:53,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:54,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:54,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:55,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:56,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:57,670 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:53:57,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:58,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:53:59,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:53:59,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:00,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:01,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:02,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:03,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:04,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:04,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:05,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:06,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:07,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:08,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:09,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:09,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:10,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:11,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:12,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:13,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:14,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:14,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:15,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:16,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:17,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:18,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:19,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:19,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:20,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:21,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:22,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:54:22,168 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:54:22,168 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:54:22,169 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:54:22,639 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:54:22,639 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:54:22,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:23,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:24,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:25,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:25,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:26,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:27,674 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:54:27,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:28,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:29,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:30,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:30,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:31,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:32,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:33,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:34,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:35,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:35,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:36,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:37,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:38,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:39,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:40,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:40,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:41,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:42,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:43,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:44,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:45,687 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:45,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:46,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:47,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:48,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:49,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:50,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:50,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:51,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:54:51,098 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:54:51,098 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:54:51,099 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:54:51,667 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:54:51,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:52,668 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:54:52,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:53,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:54,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:55,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:56,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:54:56,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:57,679 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:54:57,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:58,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:54:59,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:00,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:01,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:01,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:02,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:03,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:04,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:05,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:06,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:06,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:07,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:08,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:09,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:10,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:11,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:11,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:12,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:13,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:14,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:15,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:16,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:16,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:17,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:18,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:19,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:20,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:21,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:21,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:22,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:55:22,070 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:55:22,071 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:55:22,072 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:55:22,698 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:55:22,698 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:55:22,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:23,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:24,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:25,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:26,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:27,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:27,682 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:55:27,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:28,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:29,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:30,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:31,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:32,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:32,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:33,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:34,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:35,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:36,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:37,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:37,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:38,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:39,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:40,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:41,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:42,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:42,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:43,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:44,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:45,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:46,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:47,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:47,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:48,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:49,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:50,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:51,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:55:51,644 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:55:51,644 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:55:51,646 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:55:51,725 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:55:52,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:52,727 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:55:53,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:53,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:54,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:55,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:56,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:57,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:57,686 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:55:58,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:55:58,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:55:59,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:00,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:01,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:02,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:03,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:03,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:04,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:05,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:06,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:07,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:08,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:08,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:09,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:10,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:11,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:12,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:13,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:13,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:14,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:15,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:16,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:17,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:18,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:18,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:19,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:20,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:20,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:56:20,152 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:56:20,152 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:56:20,153 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:56:20,759 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:56:20,760 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:56:21,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:22,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:23,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:24,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:24,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:25,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:26,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:27,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:27,689 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:56:28,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:29,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:29,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:30,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:31,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:32,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:33,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:34,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:34,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:35,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:36,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:37,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:38,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:39,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:39,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:40,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:41,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:42,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:43,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:44,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:44,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:45,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:46,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:47,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:48,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:49,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:49,702 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:50,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:50,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:56:50,186 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:56:50,186 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:56:50,188 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:56:50,788 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:56:50,789 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:56:51,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:52,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:53,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:54,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:55,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:55,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:56:56,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:57,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:57,693 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:56:58,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:56:59,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:00,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:00,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:01,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:02,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:03,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:04,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:05,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:05,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:06,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:07,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:08,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:09,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:10,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:10,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:11,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:12,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:13,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:14,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:15,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:15,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:16,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:17,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:18,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:19,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:20,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:20,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:57:20,516 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:57:20,516 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:57:20,518 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:57:20,824 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:57:20,825 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:57:21,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:21,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:22,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:23,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:24,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:25,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:26,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:26,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:27,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:27,697 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:57:28,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:29,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:30,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:31,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:31,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:32,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:33,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:34,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:35,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:36,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:36,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:37,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:38,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:39,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:40,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:41,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:41,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:42,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:43,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:44,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:45,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:46,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:46,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:47,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:48,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:49,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:50,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:50,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:57:50,295 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:57:50,296 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:57:50,297 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:57:50,856 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:57:51,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:51,857 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:57:52,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:52,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:53,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:54,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:55,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:56,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:57,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:57,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:57:57,702 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:57:58,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:57:59,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:00,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:01,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:02,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:02,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:03,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:04,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:05,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:06,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:07,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:07,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:08,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:09,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:10,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:11,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:12,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:12,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:13,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:14,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:15,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:16,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:17,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:17,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:18,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:19,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:20,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:21,248 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:22,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:58:22,211 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:58:22,212 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:58:22,213 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:58:22,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:22,888 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:58:23,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:23,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:23,889 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:58:24,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:25,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:26,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:27,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:27,705 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:58:28,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:28,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:29,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:30,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:31,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:32,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:33,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:33,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:34,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:35,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:36,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:37,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:38,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:38,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:39,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:40,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:41,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:42,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:43,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:43,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:44,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:45,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:46,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:47,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:48,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:48,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:49,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:50,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:51,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:52,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:53,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:53,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:58:54,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:55,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:55,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:58:55,396 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:58:55,396 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:58:55,398 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:58:55,915 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:58:56,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:57,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:57,709 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:58:57,917 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:58:58,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:59,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:58:59,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:00,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:01,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:02,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:03,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:04,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:04,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:05,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:06,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:07,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:08,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:09,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:09,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:10,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:11,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:12,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:13,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:14,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:14,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:15,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:16,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:17,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:18,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:19,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:19,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:20,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:21,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:22,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:22,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:59:22,760 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:59:22,761 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:59:22,763 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:59:22,945 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:59:23,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:23,945 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:59:24,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:24,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:25,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:26,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:27,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:27,713 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:59:28,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:29,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:30,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:30,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:31,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:32,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:33,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:34,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:35,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:35,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:36,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:37,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:38,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:39,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:40,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:40,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:41,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:42,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:43,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:44,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:45,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:45,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:46,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:47,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:48,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:49,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:50,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:50,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:51,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:52,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:53,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:54,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:55,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 11:59:55,255 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 11:59:55,256 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 11:59:55,257 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 11:59:55,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:55,971 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 11:59:56,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 11:59:56,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:57,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:57,717 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 11:59:57,973 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 11:59:58,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 11:59:59,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:00,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:01,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:01,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:02,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:03,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:04,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:05,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:06,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:06,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:07,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:08,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:09,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:10,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:11,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:11,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:12,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:13,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:14,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:15,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:16,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:16,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:17,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:18,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:19,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:20,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:21,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:21,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:22,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:23,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:23,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:00:23,827 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:00:23,828 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:00:23,829 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:00:23,997 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:00:24,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:25,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:25,998 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:00:26,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:26,868 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:27,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:27,720 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:00:28,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:29,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:30,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:31,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:32,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:32,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:33,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:34,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:35,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:36,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:37,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:37,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:38,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:39,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:40,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:41,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:42,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:42,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:43,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:44,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:45,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:46,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:47,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:47,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:48,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:49,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:50,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:51,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:52,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:00:52,285 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:00:52,286 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:00:52,287 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:00:52,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:53,028 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:00:53,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:53,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:54,029 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:00:54,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:55,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:56,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:57,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:57,724 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:00:58,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:00:58,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:00:59,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:00,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:01,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:02,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:03,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:03,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:04,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:05,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:06,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:07,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:08,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:08,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:09,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:10,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:11,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:12,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:13,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:13,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:14,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:15,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:16,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:17,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:18,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:18,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:19,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:20,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:21,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:22,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:23,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:23,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:24,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:01:24,285 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:01:24,285 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:01:24,287 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:01:24,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:25,057 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:01:25,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:26,057 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:01:26,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:27,728 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:01:27,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:28,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:29,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:29,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:30,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:31,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:32,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:33,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:34,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:34,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:35,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:36,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:37,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:38,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:39,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:39,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:40,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:41,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:42,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:43,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:44,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:44,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:45,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:46,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:47,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:48,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:49,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:49,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:50,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:51,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:52,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:53,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:54,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:01:54,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:55,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:56,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:57,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:01:57,089 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:01:57,090 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:01:57,091 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:01:57,100 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:01:57,732 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:01:57,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:58,102 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:01:58,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:01:59,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:00,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:00,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:01,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:02,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:03,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:04,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:05,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:05,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:06,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:07,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:08,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:09,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:10,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:10,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:11,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:12,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:13,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:14,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:15,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:15,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:16,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:17,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:18,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:19,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:20,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:20,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:21,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:22,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:23,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:24,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:25,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:25,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:26,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:27,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:02:27,039 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:02:27,039 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:02:27,041 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:02:27,127 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:02:27,736 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:02:27,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:28,127 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:02:28,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:29,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:30,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:31,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:31,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:32,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:33,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:34,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:35,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:36,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:36,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:37,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:38,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:39,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:40,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:41,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:41,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:42,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:43,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:44,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:45,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:46,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:46,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:47,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:48,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:49,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:50,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:51,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:51,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:52,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:53,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:54,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:55,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:56,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:02:56,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:57,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:02:57,345 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:02:57,345 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:02:57,347 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:02:57,741 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:02:57,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:58,157 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:02:58,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:02:59,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:00,159 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:03:00,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:01,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:02,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:02,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:03,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:04,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:05,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:06,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:07,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:07,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:08,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:09,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:10,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:11,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:12,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:12,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:13,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:14,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:15,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:16,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:17,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:17,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:18,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:19,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:20,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:21,870 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:22,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:22,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:23,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:24,870 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:25,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:26,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:26,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:03:26,893 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:03:26,893 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:03:26,894 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:03:27,189 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:03:27,744 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:03:27,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:28,191 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:03:28,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:28,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:29,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:30,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:31,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:32,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:33,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:33,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:34,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:35,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:36,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:37,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:38,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:38,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:39,876 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:40,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:41,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:42,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:43,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:43,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:44,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:45,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:46,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:47,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:48,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:48,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:49,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:50,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:51,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:52,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:53,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:53,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:54,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:55,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:56,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:57,758 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:03:57,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:58,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:03:58,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:03:58,849 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:03:58,849 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:03:58,850 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:03:59,217 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:03:59,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:03:59,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:00,218 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:04:00,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:01,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:02,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:03,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:03,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:04,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:05,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:06,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:07,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:08,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:08,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:09,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:10,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:11,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:12,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:13,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:13,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:14,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:15,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:16,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:17,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:18,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:18,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:19,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:20,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:21,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:22,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:23,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:23,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:24,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:25,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:26,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:04:26,994 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:04:26,994 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:04:26,996 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:04:27,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:27,243 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:04:27,755 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:04:27,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:28,245 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:04:28,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:29,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:29,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:30,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:31,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:32,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:33,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:34,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:34,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:35,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:36,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:37,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:38,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:39,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:39,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:40,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:41,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:42,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:43,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:44,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:44,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:45,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:46,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:47,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:48,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:49,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:49,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:50,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:51,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:53,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:54,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:54,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:04:55,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:56,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:57,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:57,758 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:04:58,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:58,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:04:58,292 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:04:58,292 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:04:58,293 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:04:59,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:04:59,266 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:05:00,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:00,268 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:05:00,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:01,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:02,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:03,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:04,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:05,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:05,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:06,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:07,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:08,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:09,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:10,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:10,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:11,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:12,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:13,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:14,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:15,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:15,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:16,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:17,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:18,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:19,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:20,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:20,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:21,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:22,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:23,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:24,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:25,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:25,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:26,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:27,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:27,759 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:05:28,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:28,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:05:28,991 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:05:28,991 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:05:28,992 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:05:29,302 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:05:29,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:30,303 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:05:30,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:31,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:31,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:32,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:33,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:34,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:35,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:36,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:36,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:37,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:38,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:39,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:40,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:41,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:41,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:42,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:43,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:44,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:45,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:46,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:46,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:47,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:48,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:49,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:50,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:51,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:51,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:52,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:53,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:54,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:55,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:56,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:05:56,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:57,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:57,768 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:05:58,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:05:59,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:00,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:00,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:06:00,574 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:06:00,574 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:06:00,576 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:06:01,327 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:06:01,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:01,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:02,328 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:06:02,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:03,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:04,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:05,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:06,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:06,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:07,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:08,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:09,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:10,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:11,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:11,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:12,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:13,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:14,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:15,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:16,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:16,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:17,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:18,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:19,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:20,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:21,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:21,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:22,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:23,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:24,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:25,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:26,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:26,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:27,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:27,772 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:06:28,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:29,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:30,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:31,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:31,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:32,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:33,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:34,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:34,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:06:34,864 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:06:34,864 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:06:34,866 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:06:35,356 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:06:35,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:36,357 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:06:36,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:36,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:37,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:38,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:39,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:40,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:41,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:41,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:42,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:43,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:44,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:45,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:46,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:46,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:47,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:48,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:49,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:50,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:51,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:51,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:52,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:53,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:54,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:55,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:56,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:56,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:06:57,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:57,776 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:06:58,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:06:59,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:00,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:01,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:02,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:02,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:03,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:04,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:05,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:07:05,397 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:07:05,397 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:07:05,398 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:07:05,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:06,384 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:07:06,385 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:07:06,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:07,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:08,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:08,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:09,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:10,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:11,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:12,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:13,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:13,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:14,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:15,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:16,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:17,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:18,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:18,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:19,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:20,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:21,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:22,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:23,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:23,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:24,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:25,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:26,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:27,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:27,780 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:07:28,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:28,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:29,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:30,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:31,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:32,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:33,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:33,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:34,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:35,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:36,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:37,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:07:37,194 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:07:37,194 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:07:37,196 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:07:37,416 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:07:37,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:38,417 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:07:38,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:39,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:39,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:40,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:41,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:42,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:43,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:44,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:44,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:45,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:46,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:47,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:48,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:49,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:49,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:50,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:51,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:52,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:53,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:54,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:07:54,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:55,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:56,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:57,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:57,783 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:07:58,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:59,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:07:59,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:00,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:01,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:02,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:03,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:04,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:04,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:05,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:06,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:08:06,292 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:08:06,293 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:08:06,294 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:08:06,444 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:08:06,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:07,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:08,445 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:08:08,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:09,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:10,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:10,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:11,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:12,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:13,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:14,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:15,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:15,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:16,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:17,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:18,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:19,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:20,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:20,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:21,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:22,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:23,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:24,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:25,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:25,694 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:26,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:27,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:27,787 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:08:28,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:29,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:30,702 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:30,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:31,703 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:32,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:33,703 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:34,703 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:35,703 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:35,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:36,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:08:36,558 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:08:36,559 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:08:36,560 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:08:36,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:37,471 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:08:37,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:38,472 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:08:38,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:39,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:40,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:41,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:41,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:42,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:43,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:44,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:45,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:46,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:46,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:47,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:48,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:49,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:50,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:51,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:51,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:52,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:53,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:54,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:55,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:56,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:08:56,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:57,791 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:08:57,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:58,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:08:59,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:00,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:01,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:01,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:02,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:03,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:04,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:05,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:06,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:09:06,224 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:09:06,224 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:09:06,225 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:09:06,500 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:09:06,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:07,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:07,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:08,501 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:09:08,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:09,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:10,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:11,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:12,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:12,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:13,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:14,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:15,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:16,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:17,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:17,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:18,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:19,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:20,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:21,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:22,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:22,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:23,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:24,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:25,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:26,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:27,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:27,805 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:09:27,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:28,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:29,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:30,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:31,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:32,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:32,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:33,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:09:33,621 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:09:33,622 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:09:33,623 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:09:34,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:34,528 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:09:34,529 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:09:35,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:36,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:37,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:38,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:38,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:39,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:40,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:41,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:42,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:43,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:43,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:44,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:45,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:46,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:47,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:48,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:48,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:49,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:50,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:51,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:52,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:53,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:53,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:54,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:55,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:56,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:57,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:57,798 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:09:58,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:09:58,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:09:59,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:00,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:01,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:02,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:03,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:03,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:04,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:05,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:06,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:10:06,060 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:10:06,060 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:10:06,064 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:10:06,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:06,550 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:10:07,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:08,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:08,551 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:10:09,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:09,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:10,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:11,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:12,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:13,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:14,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:14,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:15,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:16,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:17,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:18,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:19,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:19,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:20,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:21,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:22,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:23,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:24,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:24,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:25,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:26,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:27,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:27,802 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:10:28,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:29,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:29,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:30,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:31,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:32,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:33,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:34,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:34,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:35,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:36,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:36,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:10:36,899 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:10:36,900 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:10:36,902 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:10:37,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:37,579 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:10:38,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:38,581 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:10:39,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:39,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:40,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:41,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:42,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:43,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:44,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:44,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:45,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:46,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:47,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:48,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:49,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:49,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:50,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:51,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:52,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:53,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:54,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:54,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:10:55,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:56,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:57,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:57,806 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:10:58,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:10:59,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:00,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:00,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:01,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:02,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:03,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:04,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:05,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:05,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:06,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:07,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:08,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:09,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:11:09,253 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:11:09,253 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:11:09,258 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:11:09,608 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:11:09,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:10,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:10,609 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:11:11,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:11,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:12,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:13,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:14,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:15,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:16,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:16,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:17,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:18,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:19,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:20,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:21,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:21,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:22,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:23,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:24,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:25,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:26,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:26,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:27,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:27,810 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:11:28,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:29,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:30,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:31,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:31,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:32,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:33,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:34,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:35,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:36,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:36,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:37,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:38,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:38,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:11:38,935 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:11:38,935 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:11:38,937 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:11:39,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:39,634 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:11:40,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:40,635 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:11:41,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:41,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:42,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:43,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:44,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:45,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:46,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:46,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:47,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:48,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:49,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:50,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:51,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:51,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:52,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:53,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:54,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:55,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:56,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:56,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:11:57,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:57,814 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:11:58,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:11:59,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:00,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:01,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:02,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:02,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:03,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:04,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:05,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:06,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:07,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:07,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:08,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:08,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:12:08,610 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:12:08,610 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:12:08,611 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:12:08,661 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:12:09,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:10,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:10,662 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:12:11,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:12,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:13,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:13,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:14,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:15,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:16,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:17,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:18,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:18,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:19,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:20,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:21,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:22,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:23,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:23,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:24,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:25,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:26,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:27,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:27,818 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:12:28,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:28,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:29,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:30,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:31,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:32,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:34,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:34,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:34,677 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:12:39,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:41,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:12:41,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:12:42,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:42,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:44,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:49,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:12:49,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:50,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:51,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:54,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:12:57,822 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:12:58,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:12:58,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:12:58,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:12:59,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:00,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:04,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:06,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:13:07,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:13:07,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:08,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:09,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:14,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:15,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:13:15,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:13:15,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:16,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:17,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:19,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:24,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:27,826 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:13:28,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:28,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:29,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:30,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:30,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:31,026 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:13:31,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:32,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:33,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:34,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:35,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:35,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:36,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:37,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:38,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:39,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:40,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:40,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:41,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:42,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:43,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:44,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:45,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:45,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:46,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:47,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:48,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:49,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:50,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:50,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:51,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:52,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:53,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:54,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:55,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:55,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:13:56,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:57,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:57,831 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:13:58,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:13:59,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:00,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:00,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:01,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:02,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:03,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:04,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:05,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:05,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:06,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:07,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:08,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:09,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:10,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:10,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:11,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:12,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:13,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:14,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:15,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:15,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:16,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:17,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:18,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:19,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:20,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:20,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:25,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:27,834 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:14:31,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:36,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:41,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:42,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:14:42,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:14:42,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:43,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:44,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:45,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:46,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:46,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:47,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:48,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:49,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:50,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:51,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:51,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:52,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:53,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:54,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:55,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:56,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:56,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:14:57,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:57,839 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:14:58,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:14:59,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:00,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:01,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:01,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:02,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:03,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:04,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:05,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:06,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:06,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:07,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:08,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:09,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:10,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:11,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:11,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:12,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:13,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:14,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:15,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:16,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:16,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:17,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:18,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:19,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:20,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:21,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:21,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:22,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:23,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:24,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:25,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:26,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:26,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:27,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:27,843 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:15:28,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:29,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:30,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:31,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:31,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:32,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:33,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:34,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:35,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:36,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:36,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:37,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:38,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:39,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:40,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:41,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:41,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:42,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:43,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:44,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:45,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:46,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:46,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:47,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:48,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:49,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:50,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:51,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:51,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:52,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:53,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:54,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:55,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:56,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:56,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:15:57,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:57,847 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:15:58,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:15:59,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:00,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:01,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:02,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:02,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:03,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:04,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:05,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:06,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:07,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:07,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:08,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:09,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:10,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:11,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:12,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:12,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:13,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:14,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:15,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:16,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:17,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:17,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:18,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:19,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:20,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:21,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:22,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:22,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:23,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:24,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:25,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:26,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:27,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:27,851 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:16:28,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:28,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:29,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:30,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:31,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:16:33,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:38,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:43,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:48,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:53,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:16:57,894 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:16:58,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:03,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:08,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:12,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 12:17:12,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:13,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:13,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:14,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:14,961 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:17:15,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:16,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:17,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:18,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:18,963 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:17:19,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:19,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:20,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:21,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:22,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:23,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:24,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:24,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:25,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:26,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:27,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:27,857 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:17:28,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:29,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:29,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:30,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:31,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:32,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:33,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:34,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:34,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:35,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:36,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:37,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:38,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:39,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:39,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:40,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:41,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:42,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:43,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:44,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:44,869 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:45,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:46,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:46,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:17:46,576 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:17:46,576 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:17:46,581 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:17:46,990 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:17:47,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:48,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:48,991 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:17:49,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:50,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:50,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:51,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:52,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:53,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:54,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:55,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:55,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:17:56,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:57,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:57,860 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:17:58,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:17:59,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:00,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:00,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:01,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:02,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:03,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:04,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:05,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:05,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:06,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:07,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:08,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:09,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:10,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:10,868 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:11,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:12,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:13,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:14,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:15,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:15,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:18:15,766 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:18:15,766 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:18:15,767 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:18:16,017 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:18:16,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:16,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:17,018 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:18:17,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:18,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:19,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:20,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:21,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:21,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:22,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:23,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:24,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:25,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:26,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:26,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:27,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:27,864 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:18:28,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:29,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:30,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:31,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:31,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:32,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:33,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:34,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:35,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:36,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:36,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:37,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:38,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:39,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:40,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:41,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:41,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:42,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:43,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:44,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:45,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:46,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:46,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:47,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:47,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:18:47,696 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:18:47,696 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:18:47,701 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:18:48,041 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:18:48,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:49,042 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:18:49,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:50,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:51,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:52,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:52,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:53,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:54,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:55,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:56,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:57,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:57,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:18:57,867 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:18:58,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:18:59,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:00,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:01,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:02,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:02,870 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:03,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:04,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:05,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:06,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:07,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:07,871 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:08,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:09,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:10,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:11,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:12,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:12,871 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:13,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:14,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:15,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:16,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:17,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:17,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:19:17,855 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:19:17,855 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:19:17,860 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:19:17,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:18,072 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:19:18,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:19,073 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:19:19,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:20,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:21,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:22,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:22,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:23,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:24,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:25,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:26,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:27,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:27,871 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:19:28,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:28,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:29,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:30,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:31,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:32,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:33,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:33,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:34,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:35,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:36,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:37,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:38,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:38,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:39,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:40,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:41,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:42,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:43,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:43,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:44,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:45,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:46,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:47,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:48,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:48,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:49,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:49,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:19:49,326 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:19:49,327 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:19:49,332 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:19:50,098 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:19:50,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:51,100 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:19:51,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:52,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:53,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:54,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:54,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:19:55,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:56,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:57,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:57,875 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:19:58,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:59,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:19:59,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:00,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:01,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:02,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:03,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:04,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:04,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:05,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:06,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:07,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:08,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:09,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:09,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:10,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:11,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:12,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:13,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:14,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:14,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:15,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:16,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:17,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:18,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:19,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:19,888 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:20,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:21,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:22,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:22,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:20:22,498 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:20:22,498 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:20:22,503 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:20:23,133 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:20:23,134 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:20:23,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:24,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:25,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:25,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:26,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:27,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:27,880 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:20:28,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:29,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:30,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:30,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:31,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:32,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:33,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:34,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:35,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:35,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:36,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:37,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:38,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:39,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:40,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:40,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:41,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:42,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:43,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:44,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:45,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:45,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:46,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:47,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:48,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:49,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:50,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:50,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:51,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:52,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:53,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:54,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:55,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:55,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:20:55,856 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:20:55,857 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:20:55,862 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:20:55,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:20:56,165 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:20:56,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:57,166 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:20:57,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:57,883 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:20:58,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:20:59,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:00,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:01,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:01,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:02,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:03,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:04,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:05,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:06,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:06,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:07,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:08,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:09,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:10,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:11,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:11,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:12,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:13,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:14,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:15,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:16,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:16,888 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:17,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:18,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:19,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:20,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:21,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:21,888 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:22,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:23,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:24,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:25,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:26,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:26,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:27,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:27,892 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:21:28,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:29,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:30,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:31,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:21:31,066 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:21:31,067 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:21:31,068 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:21:31,197 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:21:31,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:32,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:32,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:33,199 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:21:33,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:34,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:35,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:36,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:37,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:37,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:38,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:39,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:40,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:41,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:42,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:42,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:43,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:44,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:45,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:46,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:47,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:47,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:48,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:49,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:50,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:51,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:52,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:52,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:53,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:54,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:55,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:56,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:57,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:21:57,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:57,894 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:21:58,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:21:59,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:00,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:01,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:22:01,396 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:22:01,397 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:22:01,398 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:22:02,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:02,230 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:22:02,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:03,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:03,231 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:22:04,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:05,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:06,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:07,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:07,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:08,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:09,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:10,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:11,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:12,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:12,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:13,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:14,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:15,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:16,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:17,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:17,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:18,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:19,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:20,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:21,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:22,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:22,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:23,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:24,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:25,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:26,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:27,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:27,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:27,900 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:22:28,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:29,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:30,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:30,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:22:30,677 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:22:30,677 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:22:30,679 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:22:31,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:31,261 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:22:31,262 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:22:32,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:32,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:33,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:34,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:35,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:36,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:37,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:37,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:38,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:39,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:40,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:41,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:42,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:42,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:43,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:44,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:45,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:46,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:47,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:47,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:48,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:49,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:50,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:51,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:52,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:52,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:53,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:54,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:55,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:56,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:57,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:57,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:22:57,904 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:22:58,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:22:59,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:00,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:01,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:01,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:23:01,399 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:23:01,399 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:23:01,401 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:23:02,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:02,292 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:23:03,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:03,292 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:23:03,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:04,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:05,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:06,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:07,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:08,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:08,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:09,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:10,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:11,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:12,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:13,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:13,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:14,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:15,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:16,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:17,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:18,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:18,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:19,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:20,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:21,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:22,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:23,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:23,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:24,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:25,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:26,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:27,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:27,912 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:23:28,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:28,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:29,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:30,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:31,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:31,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:23:31,533 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:23:31,533 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:23:31,534 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:23:32,315 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:23:32,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:33,316 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:23:33,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:34,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:34,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:35,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:36,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:37,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:38,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:39,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:39,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:40,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:41,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:42,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:43,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:44,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:44,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:45,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:46,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:47,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:48,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:49,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:49,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:50,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:51,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:52,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:53,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:54,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:54,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:23:55,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:56,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:57,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:57,909 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:23:58,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:59,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:23:59,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:00,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:01,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:02,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:03,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:24:03,436 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:24:03,436 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:24:03,441 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:24:03,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:04,343 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:24:04,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:05,344 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:24:05,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:05,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:06,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:07,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:08,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:09,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:10,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:10,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:11,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:12,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:13,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:14,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:15,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:15,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:17,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:18,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:19,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:20,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:20,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:21,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:22,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:23,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:24,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:25,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:25,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:26,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:27,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:27,913 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:24:28,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:29,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:30,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:30,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:31,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:32,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:33,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:34,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:35,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:35,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:36,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:36,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:24:36,238 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:24:36,238 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:24:36,241 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:24:36,377 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:24:37,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:38,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:38,379 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:24:39,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:40,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:41,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:41,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:42,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:43,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:44,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:45,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:46,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:46,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:47,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:48,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:49,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:50,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:51,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:51,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:52,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:53,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:54,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:55,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:56,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:56,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:24:57,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:57,918 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:24:58,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:24:59,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:00,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:01,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:01,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:02,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:03,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:04,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:05,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:06,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:06,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:07,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:07,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:25:07,947 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:25:07,947 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:25:07,950 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:25:08,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:08,405 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:25:09,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:10,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:10,406 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:25:11,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:11,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:12,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:13,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:14,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:15,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:16,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:16,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:17,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:18,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:19,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:20,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:21,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:21,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:22,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:23,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:24,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:25,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:26,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:26,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:27,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:27,921 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:25:28,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:29,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:30,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:31,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:32,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:32,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:33,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:34,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:35,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:36,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:37,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:37,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:38,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:39,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:40,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:40,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:25:40,567 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:25:40,568 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:25:40,573 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:25:41,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:41,441 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:25:42,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:42,441 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:25:43,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:43,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:44,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:45,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:46,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:47,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:48,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:48,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:49,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:50,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:51,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:52,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:53,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:53,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:54,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:55,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:56,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:57,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:57,925 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:25:58,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:25:58,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:25:59,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:00,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:01,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:02,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:03,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:03,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:04,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:05,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:06,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:07,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:08,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:08,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:09,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:09,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:26:09,562 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:26:09,563 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:26:09,564 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:26:10,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:10,470 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:26:11,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:12,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:12,471 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:26:13,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:14,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:14,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:15,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:16,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:17,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:18,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:19,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:19,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:20,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:21,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:22,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:23,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:24,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:24,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:25,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:26,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:27,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:27,929 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:26:28,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:29,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:29,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:30,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:31,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:32,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:33,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:34,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:34,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:35,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:36,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:37,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:38,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:39,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:39,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:40,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:40,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:26:40,342 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:26:40,342 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:26:40,343 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:26:40,498 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:26:41,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:42,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:42,500 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:26:43,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:44,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:45,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:45,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:46,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:47,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:48,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:49,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:50,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:50,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:51,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:52,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:53,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:54,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:55,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:55,385 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:26:56,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:57,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:57,931 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:26:58,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:26:59,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:00,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:00,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:01,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:02,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:03,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:04,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:05,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:05,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:06,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:07,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:08,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:09,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:09,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:27:09,446 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:27:09,446 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:27:09,447 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:27:09,525 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:27:10,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:11,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:11,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:12,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:12,527 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:27:13,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:14,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:15,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:16,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:16,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:17,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:18,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:19,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:20,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:21,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:21,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:22,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:23,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:24,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:25,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:26,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:26,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:27,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:27,935 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:27:28,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:29,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:30,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:31,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:31,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:32,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:33,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:34,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:35,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:36,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:36,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:37,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:38,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:39,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:40,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:27:40,191 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:27:40,192 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:27:40,193 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:27:40,554 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:27:40,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:41,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:42,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:42,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:42,556 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:27:43,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:44,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:45,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:46,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:47,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:47,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:48,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:49,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:50,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:51,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:52,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:52,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:53,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:54,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:55,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:56,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:57,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:27:57,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:57,940 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:27:58,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:27:59,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:00,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:01,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:02,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:02,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:03,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:04,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:05,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:06,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:07,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:07,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:08,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:09,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:10,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:11,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:11,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:28:11,805 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:28:11,806 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:28:11,808 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:28:12,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:12,586 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:28:13,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:13,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:14,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:14,587 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:28:15,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:16,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:17,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:18,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:18,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:19,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:20,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:21,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:22,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:23,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:23,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:24,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:25,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:26,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:27,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:27,943 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:28:28,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:28,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:29,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:30,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:31,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:32,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:33,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:33,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:34,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:35,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:36,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:37,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:38,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:38,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:39,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:40,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:41,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:42,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:28:42,254 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:28:42,254 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:28:42,257 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:28:42,614 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:28:42,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:43,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:44,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:44,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:44,616 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:28:45,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:46,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:47,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:48,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:49,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:49,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:50,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:51,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:52,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:53,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:54,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:28:54,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:55,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:56,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:57,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:57,946 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:28:58,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:59,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:28:59,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:00,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:01,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:02,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:03,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:04,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:04,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:05,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:06,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:07,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:08,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:09,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:09,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:10,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:11,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:12,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:29:12,349 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:29:12,349 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:29:12,353 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:29:12,644 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:29:12,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:13,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:14,646 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:29:14,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:15,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:15,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:16,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:17,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:18,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:19,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:20,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:20,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:21,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:22,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:23,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:24,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:25,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:25,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:26,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:27,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:27,950 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:29:28,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:29,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:30,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:30,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:31,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:32,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:33,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:34,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:35,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:35,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:36,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:37,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:38,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:39,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:40,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:40,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:41,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:42,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:29:42,689 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:29:42,690 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:29:42,695 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:29:43,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:43,668 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:29:43,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:44,669 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:29:44,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:45,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:46,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:46,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:47,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:48,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:49,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:50,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:51,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:51,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:52,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:53,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:54,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:55,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:56,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:29:56,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:57,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:57,954 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:29:58,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:29:59,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:00,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:01,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:01,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:02,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:03,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:04,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:05,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:06,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:06,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:07,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:08,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:09,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:10,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:11,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:11,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:12,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:13,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:14,838 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:30:14,839 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:30:14,839 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:30:14,844 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:30:14,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:15,706 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:30:15,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:16,707 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:30:16,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:17,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:17,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:18,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:19,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:20,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:21,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:22,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:22,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:23,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:24,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:25,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:26,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:27,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:27,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:27,958 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:30:29,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:29,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:30,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:31,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:32,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:32,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:33,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:34,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:35,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:36,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:37,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:37,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:38,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:39,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:40,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:41,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:42,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:42,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:43,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:44,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:45,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:46,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:47,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:30:47,619 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:30:47,620 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:30:47,623 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:30:47,733 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:30:48,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:48,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:48,734 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:30:49,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:50,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:51,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:52,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:53,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:53,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:54,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:55,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:56,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:57,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:57,963 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:30:58,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:30:58,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:30:59,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:00,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:01,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:02,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:03,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:03,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:04,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:05,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:06,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:07,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:08,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:08,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:09,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:10,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:11,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:12,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:13,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:13,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:14,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:15,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:16,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:17,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:18,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:18,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:19,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:20,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:21,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:22,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:22,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:31:22,276 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:31:22,276 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:31:22,279 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:31:22,762 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:31:23,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:24,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:24,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:24,763 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:31:25,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:26,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:27,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:27,964 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:31:28,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:29,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:29,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:30,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:31,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:32,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:33,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:34,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:34,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:35,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:36,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:37,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:38,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:39,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:39,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:40,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:41,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:42,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:43,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:44,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:44,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:45,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:46,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:47,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:48,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:49,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:49,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:50,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:51,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:52,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:53,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:53,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:31:53,376 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:31:53,377 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:31:53,379 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:31:53,787 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:31:54,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:54,788 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:31:55,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:55,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:31:56,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:57,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:57,969 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:31:58,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:31:59,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:00,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:00,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:01,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:02,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:03,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:04,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:05,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:05,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:06,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:07,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:08,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:09,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:10,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:10,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:11,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:12,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:13,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:14,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:15,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:15,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:16,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:17,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:18,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:19,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:20,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:20,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:21,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:21,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:32:21,482 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:32:21,482 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:32:21,484 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:32:21,810 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:32:22,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:22,811 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:32:23,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:24,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:25,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:26,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:26,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:27,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:27,973 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:32:28,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:29,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:30,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:31,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:31,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:32,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:33,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:34,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:35,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:36,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:36,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:37,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:38,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:39,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:40,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:41,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:41,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:42,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:43,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:44,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:45,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:46,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:46,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:47,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:48,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:49,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:50,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:51,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:51,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:52,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:53,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:53,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:32:53,663 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:32:53,664 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:32:53,665 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:32:53,853 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:32:54,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:54,854 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:32:55,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:56,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:57,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:57,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:32:57,976 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:32:58,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:32:59,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:00,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:01,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:02,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:02,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:03,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:04,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:05,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:06,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:07,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:07,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:08,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:09,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:10,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:11,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:12,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:12,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:13,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:14,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:15,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:16,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:17,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:17,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:18,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:19,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:20,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:21,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:22,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:22,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:23,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:24,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:33:24,298 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:33:24,298 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:33:24,299 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:33:24,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:24,884 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:33:25,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:26,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:26,886 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:33:27,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:27,981 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:33:28,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:28,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:29,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:30,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:31,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:32,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:33,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:33,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:34,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:35,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:36,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:37,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:38,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:38,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:39,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:40,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:41,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:42,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:43,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:43,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:44,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:45,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:46,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:47,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:48,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:48,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:49,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:50,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:51,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:52,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:53,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:33:53,322 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:33:53,322 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:33:53,324 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:33:53,910 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:33:53,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:54,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:33:54,910 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:33:54,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:55,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:56,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:57,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:57,984 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:33:58,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:59,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:33:59,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:00,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:01,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:02,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:03,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:04,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:04,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:05,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:06,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:07,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:08,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:09,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:09,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:10,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:11,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:12,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:13,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:14,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:14,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:15,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:16,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:17,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:18,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:19,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:19,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:20,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:21,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:22,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:23,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:24,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:25,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:25,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:34:25,608 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:34:25,609 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:34:25,610 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:34:25,938 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:34:26,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:26,939 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:34:27,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:27,988 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:34:28,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:29,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:30,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:30,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:31,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:32,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:33,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:34,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:35,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:35,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:36,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:37,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:38,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:39,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:40,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:40,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:41,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:42,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:43,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:44,169 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:45,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:45,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:46,169 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:47,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:48,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:49,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:50,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:51,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:51,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:52,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:52,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:34:52,303 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:34:52,303 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:34:52,305 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:34:52,964 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:34:52,965 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:34:53,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:54,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:55,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:56,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:56,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:34:57,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:57,992 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:34:58,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:34:59,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:00,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:01,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:01,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:02,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:03,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:04,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:05,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:06,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:06,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:07,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:08,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:09,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:10,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:11,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:11,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:12,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:13,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:14,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:15,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:16,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:16,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:17,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:18,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:19,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:20,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:21,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:22,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:22,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:23,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:24,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:24,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:35:24,392 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:35:24,393 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:35:24,394 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:35:24,992 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:35:25,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:26,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:26,993 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:35:27,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:27,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:27,996 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:35:28,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:29,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:30,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:31,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:32,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:32,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:33,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:34,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:35,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:36,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:37,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:38,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:38,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:39,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:40,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:41,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:42,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:43,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:43,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:44,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:45,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:46,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:47,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:48,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:48,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:49,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:50,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:51,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:51,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:35:51,426 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:35:51,427 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:35:51,428 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:35:52,027 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:35:52,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:53,028 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:35:53,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:53,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:54,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:55,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:56,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:57,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:58,001 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:35:58,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:35:59,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:35:59,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:00,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:01,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:02,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:03,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:04,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:04,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:05,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:06,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:07,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:08,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:09,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:09,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:10,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:11,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:12,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:13,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:14,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:14,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:15,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:16,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:17,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:18,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:19,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:19,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:20,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:21,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:36:21,144 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:36:21,144 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:36:21,146 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:36:21,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:22,050 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:36:22,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:23,051 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:36:23,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:24,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:24,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:25,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:26,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:27,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:28,004 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:36:28,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:29,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:30,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:30,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:31,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:32,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:33,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:34,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:35,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:35,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:36,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:37,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:38,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:39,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:40,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:40,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:41,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:42,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:43,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:44,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:45,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:45,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:46,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:47,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:48,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:48,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:36:48,974 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:36:48,974 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:36:48,976 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:36:49,074 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:36:49,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:50,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:50,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:51,076 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:36:51,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:52,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:53,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:54,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:55,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:36:55,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:56,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:57,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:58,008 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:36:58,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:36:59,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:00,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:01,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:01,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:02,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:03,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:04,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:05,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:06,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:06,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:07,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:08,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:09,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:10,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:11,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:11,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:12,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:13,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:14,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:15,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:16,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:16,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:17,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:18,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:19,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:20,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:20,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:37:20,729 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:37:20,729 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:37:20,730 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:37:21,108 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:37:21,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:21,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:22,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:23,109 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:37:23,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:24,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:25,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:26,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:26,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:27,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:28,012 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:37:28,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:29,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:30,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:31,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:32,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:32,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:33,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:34,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:35,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:36,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:37,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:37,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:38,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:39,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:40,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:41,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:42,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:42,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:43,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:44,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:45,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:46,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:47,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:47,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:48,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:49,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:49,876 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:37:49,877 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:37:49,877 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:37:49,878 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:37:50,136 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:37:50,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:51,137 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:37:51,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:52,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:52,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:53,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:54,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:55,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:56,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:57,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:57,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:37:58,016 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:37:58,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:37:59,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:00,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:01,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:02,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:03,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:03,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:04,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:05,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:06,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:07,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:08,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:08,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:09,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:10,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:11,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:12,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:13,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:13,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:14,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:15,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:16,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:17,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:18,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:18,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:19,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:38:19,403 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:38:19,403 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:38:19,404 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:38:19,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:20,164 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:38:20,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:21,165 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:38:21,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:22,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:23,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:23,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:24,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:25,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:26,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:27,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:28,020 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:38:28,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:29,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:29,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:30,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:31,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:32,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:33,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:34,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:34,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:35,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:36,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:37,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:38,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:39,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:39,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:40,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:41,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:42,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:43,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:44,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:44,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:45,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:46,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:47,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:48,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:49,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:49,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:50,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:38:50,805 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:38:50,805 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:38:50,808 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:38:51,190 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:38:51,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:51,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:52,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:53,192 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:38:53,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:54,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:38:54,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:55,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:56,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:57,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:58,025 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:38:58,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:38:59,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:00,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:00,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:01,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:02,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:03,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:04,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:05,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:05,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:06,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:07,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:08,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:09,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:10,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:10,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:11,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:12,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:13,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:14,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:15,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:15,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:16,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:17,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:18,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:19,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:20,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:20,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:21,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:39:21,627 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:39:21,627 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:39:21,628 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:39:22,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:22,212 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:39:23,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:23,213 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:39:24,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:25,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:25,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:26,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:27,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:28,028 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:39:28,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:29,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:30,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:31,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:31,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:32,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:33,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:34,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:35,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:36,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:36,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:37,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:38,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:39,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:40,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:41,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:41,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:42,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:43,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:44,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:45,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:46,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:46,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:47,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:48,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:49,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:50,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:51,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:51,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:51,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:39:51,442 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:39:51,443 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:39:51,444 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:39:52,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:52,236 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:39:53,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:53,237 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:39:54,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:55,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:56,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:56,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:39:57,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:58,031 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:39:58,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:39:59,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:00,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:01,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:02,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:02,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:03,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:04,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:05,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:06,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:07,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:07,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:08,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:09,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:10,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:11,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:12,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:12,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:13,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:14,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:15,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:16,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:17,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:17,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:18,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:19,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:20,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:21,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:22,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:22,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:23,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:24,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:25,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:26,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:26,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:40:26,771 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:40:26,772 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:40:26,773 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:40:27,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:27,272 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:40:27,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:28,035 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:40:28,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:29,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:29,273 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:40:30,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:31,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:32,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:33,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:33,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:34,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:35,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:36,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:37,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:38,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:38,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:39,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:40,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:41,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:42,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:43,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:43,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:44,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:45,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:46,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:47,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:48,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:48,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:49,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:50,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:51,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:52,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:53,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:53,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:54,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:55,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:56,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:57,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:57,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:40:57,658 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:40:57,659 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:40:57,660 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:40:58,039 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:40:58,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:40:58,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:58,295 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:40:59,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:40:59,296 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:41:00,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:01,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:02,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:03,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:03,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:04,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:05,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:06,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:07,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:08,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:08,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:09,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:10,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:11,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:12,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:13,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:13,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:14,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:15,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:16,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:17,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:18,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:18,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:19,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:20,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:21,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:22,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:23,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:23,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:24,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:25,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:26,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:26,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:41:26,652 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:41:26,652 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:41:26,653 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:41:27,323 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:41:27,323 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:41:27,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:28,044 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:41:28,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:29,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:29,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:30,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:31,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:32,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:33,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:34,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:34,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:35,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:36,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:37,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:38,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:39,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:39,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:40,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:41,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:42,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:43,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:44,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:44,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:45,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:46,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:47,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:48,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:49,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:49,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:50,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:51,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:52,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:53,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:54,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:41:54,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:55,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:56,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:57,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:41:57,177 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:41:57,177 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:41:57,179 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:41:57,353 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:41:57,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:58,047 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:41:58,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:41:59,354 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:41:59,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:00,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:00,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:01,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:02,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:03,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:04,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:05,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:05,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:06,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:07,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:08,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:09,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:10,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:10,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:11,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:12,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:13,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:14,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:15,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:15,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:16,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:17,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:18,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:19,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:20,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:20,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:21,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:22,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:23,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:24,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:25,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:25,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:26,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:27,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:28,051 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:42:28,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:42:28,266 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:42:28,266 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:42:28,267 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:42:28,384 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:42:28,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:29,385 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:42:29,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:30,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:30,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:31,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:32,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:33,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:34,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:35,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:35,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:36,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:37,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:38,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:39,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:40,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:40,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:41,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:42,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:43,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:44,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:45,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:45,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:46,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:47,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:48,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:49,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:50,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:50,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:51,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:52,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:53,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:54,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:55,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:42:55,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:56,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:57,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:58,058 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:42:58,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:42:58,392 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:42:58,393 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:42:58,395 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:42:58,411 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:42:58,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:42:59,412 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:42:59,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:00,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:00,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:01,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:02,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:03,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:04,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:05,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:05,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:06,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:07,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:08,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:09,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:10,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:10,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:11,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:12,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:13,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:14,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:15,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:15,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:16,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:17,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:18,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:19,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:20,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:20,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:21,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:22,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:23,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:24,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:25,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:25,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:26,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:27,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:43:27,802 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:43:27,803 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:43:27,804 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:43:28,059 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:43:28,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:28,436 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:43:28,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:29,437 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:43:29,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:30,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:31,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:31,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:32,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:33,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:34,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:35,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:36,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:36,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:37,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:38,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:39,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:40,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:41,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:41,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:42,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:43,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:44,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:45,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:46,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:46,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:47,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:48,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:49,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:50,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:51,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:51,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:52,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:53,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:54,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:55,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:56,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:43:56,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:57,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:43:57,896 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:43:57,896 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:43:57,898 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:43:58,063 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:43:58,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:58,462 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:43:59,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:43:59,463 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:44:00,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:01,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:02,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:02,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:03,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:04,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:05,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:06,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:07,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:07,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:08,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:09,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:10,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:11,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:12,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:12,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:13,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:14,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:15,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:16,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:17,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:17,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:18,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:19,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:20,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:21,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:22,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:22,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:23,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:24,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:25,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:26,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:27,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:27,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:28,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:28,066 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:44:29,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:30,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:31,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:32,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:32,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:44:32,860 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:44:32,861 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:44:32,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:32,862 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:44:33,502 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:44:33,503 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:44:33,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:34,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:35,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:36,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:37,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:37,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:38,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:39,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:40,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:41,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:42,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:42,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:43,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:44,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:45,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:46,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:47,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:47,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:48,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:49,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:50,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:51,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:52,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:52,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:53,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:54,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:55,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:56,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:57,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:57,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:44:58,070 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:44:58,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:44:59,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:00,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:01,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:02,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:03,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:03,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:04,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:05,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:45:05,507 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:45:05,507 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:45:05,509 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:45:05,535 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:45:05,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:06,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:07,537 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:45:07,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:08,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:08,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:09,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:10,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:11,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:12,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:13,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:13,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:14,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:15,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:16,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:17,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:18,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:18,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:19,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:20,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:21,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:22,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:23,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:23,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:24,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:25,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:26,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:27,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:28,074 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:45:28,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:29,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:29,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:30,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:31,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:32,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:33,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:34,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:34,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:35,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:36,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:37,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:45:37,673 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:45:37,674 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:45:37,675 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:45:38,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:38,563 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:45:39,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:39,565 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:45:39,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:40,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:41,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:42,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:43,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:44,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:44,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:45,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:46,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:47,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:48,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:49,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:49,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:50,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:51,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:52,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:53,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:54,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:54,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:45:55,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:56,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:57,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:58,078 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:45:58,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:45:59,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:00,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:00,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:01,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:02,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:03,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:04,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:05,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:05,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:06,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:07,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:08,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:09,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:09,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:46:09,620 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:46:09,620 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:46:09,621 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:46:10,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:10,595 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:46:10,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:11,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:11,596 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:46:12,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:13,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:14,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:15,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:15,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:16,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:17,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:18,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:19,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:20,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:20,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:21,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:22,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:23,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:24,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:25,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:25,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:26,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:27,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:28,082 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:46:28,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:29,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:30,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:31,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:31,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:32,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:33,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:34,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:35,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:36,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:36,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:37,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:38,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:39,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:39,642 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:46:39,643 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:46:39,644 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:46:39,645 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:46:40,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:40,631 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:46:41,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:41,632 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:46:41,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:42,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:43,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:44,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:45,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:46,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:46,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:47,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:48,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:49,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:50,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:51,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:51,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:52,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:53,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:54,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:55,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:56,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:56,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:46:57,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:58,086 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:46:58,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:46:59,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:00,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:01,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:02,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:02,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:03,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:04,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:05,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:06,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:07,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:07,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:08,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:09,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:10,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:11,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:12,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:12,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:13,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:13,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:47:13,338 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:47:13,339 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:47:13,341 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:47:13,660 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:47:14,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:15,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:15,663 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:47:16,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:17,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:17,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:18,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:19,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:20,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:21,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:22,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:22,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:23,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:24,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:25,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:26,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:27,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:27,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:28,090 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:47:28,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:29,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:30,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:31,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:32,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:33,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:33,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:34,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:35,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:36,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:37,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:38,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:38,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:39,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:40,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:40,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:47:40,268 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:47:40,268 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:47:40,270 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:47:40,688 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:47:41,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:42,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:42,690 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:47:43,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:43,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:44,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:45,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:46,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:47,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:48,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:48,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:49,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:50,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:51,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:52,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:53,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:53,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:54,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:55,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:56,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:57,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:58,094 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:47:58,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:47:59,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:47:59,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:00,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:01,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:02,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:03,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:04,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:04,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:05,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:06,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:07,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:08,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:09,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:09,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:09,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:48:09,622 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:48:09,623 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:48:09,625 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:48:09,718 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:48:10,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:11,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:12,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:12,720 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:48:13,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:14,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:14,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:15,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:16,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:17,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:18,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:19,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:19,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:20,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:21,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:22,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:23,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:24,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:24,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:25,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:26,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:27,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:28,098 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:48:28,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:29,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:30,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:30,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:31,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:32,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:33,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:34,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:35,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:35,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:36,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:37,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:38,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:39,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:39,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:48:39,488 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:48:39,489 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:48:39,490 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:48:39,747 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:48:40,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:40,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:40,748 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:48:41,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:42,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:43,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:44,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:45,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:45,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:46,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:47,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:48,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:49,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:50,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:50,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:51,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:52,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:53,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:54,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:55,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:55,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:48:56,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:57,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:58,102 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:48:58,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:48:59,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:00,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:01,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:01,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:02,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:03,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:04,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:05,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:06,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:06,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:07,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:08,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:08,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:49:08,501 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:49:08,501 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:49:08,503 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:49:08,778 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:49:09,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:10,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:10,779 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:49:11,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:11,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:12,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:13,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:14,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:15,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:16,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:16,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:17,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:18,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:19,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:20,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:21,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:21,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:22,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:23,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:24,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:25,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:26,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:26,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:27,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:28,107 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:49:28,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:29,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:30,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:31,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:32,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:32,270 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:33,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:34,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:35,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:36,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:37,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:37,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:38,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:38,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:49:38,279 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:49:38,280 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:49:38,281 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:49:38,806 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:49:39,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:40,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:40,807 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:49:41,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:42,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:42,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:43,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:44,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:45,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:46,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:47,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:47,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:48,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:49,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:50,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:51,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:52,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:52,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:53,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:54,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:55,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:56,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:57,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:57,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:49:58,110 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:49:58,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:49:59,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:00,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:01,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:02,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:03,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:03,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:04,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:05,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:06,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:07,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:08,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:08,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:08,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:50:08,757 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:50:08,757 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:50:08,758 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:50:08,840 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:50:09,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:10,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:10,842 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:50:11,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:12,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:13,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:13,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:14,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:15,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:16,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:17,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:18,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:18,797 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:19,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:20,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:21,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:22,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:23,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:23,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:24,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:25,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:26,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:27,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:28,114 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:50:28,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:29,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:29,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:30,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:31,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:32,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:33,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:34,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:34,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:35,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:36,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:37,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:50:37,065 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:50:37,065 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:50:37,066 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:50:37,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:37,871 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:50:38,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:38,872 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:50:39,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:40,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:40,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:41,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:42,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:43,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:44,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:45,105 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:45,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:46,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:47,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:48,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:49,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:50,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:50,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:51,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:52,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:53,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:54,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:55,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:50:55,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:56,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:57,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:58,118 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:50:58,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:50:59,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:00,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:00,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:01,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:02,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:03,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:04,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:05,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:05,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:06,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:06,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:51:06,549 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:51:06,549 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:51:06,551 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:51:06,894 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:51:07,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:08,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:08,896 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:51:09,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:10,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:10,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:11,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:12,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:13,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:14,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:15,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:15,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:16,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:17,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:18,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:19,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:20,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:20,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:21,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:22,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:23,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:24,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:25,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:25,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:26,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:27,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:28,127 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:51:28,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:29,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:30,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:31,137 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:31,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:32,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:33,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:34,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:51:34,132 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:51:34,132 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:51:34,134 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:51:34,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:34,927 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:51:35,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:36,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:36,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:36,928 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:51:37,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:38,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:39,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:40,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:41,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:41,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:42,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:43,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:44,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:45,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:46,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:46,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:47,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:48,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:49,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:50,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:51,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:51,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:52,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:53,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:54,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:55,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:56,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:51:56,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:57,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:58,123 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:51:58,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:51:59,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:00,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:01,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:02,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:02,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:03,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:04,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:05,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:06,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:52:06,351 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:52:06,351 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:52:06,355 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:52:06,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:06,954 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:52:07,391 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:07,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:08,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:08,956 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:52:09,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:10,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:11,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:12,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:12,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:13,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:14,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:15,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:16,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:17,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:17,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:18,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:19,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:20,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:21,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:22,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:22,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:23,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:24,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:25,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:26,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:27,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:27,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:28,128 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:52:28,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:29,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:30,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:31,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:32,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:33,137 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:33,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:34,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:35,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:36,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:37,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:52:37,176 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:52:37,176 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:52:37,181 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:52:37,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:38,000 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:52:38,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:38,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:39,001 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:52:39,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:40,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:41,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:42,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:43,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:43,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:44,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:45,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:46,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:47,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:48,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:48,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:49,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:50,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:51,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:52,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:53,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:53,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:54,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:55,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:56,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:57,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:58,131 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:52:58,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:52:59,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:52:59,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:00,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:01,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:02,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:03,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:04,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:04,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:05,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:06,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:07,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:08,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:53:08,025 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:53:08,026 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:53:08,028 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:53:08,028 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:53:08,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:09,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:10,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:10,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:11,030 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:53:11,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:12,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:13,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:14,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:15,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:15,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:16,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:17,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:18,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:19,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:20,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:20,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:21,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:22,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:23,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:24,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:25,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:25,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:26,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:27,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:28,135 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:53:28,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:29,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:30,137 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:30,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:31,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:32,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:33,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:34,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:35,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:35,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:36,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:37,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:37,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:53:37,862 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:53:37,862 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:53:37,863 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:53:38,059 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:53:38,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:39,060 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:53:39,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:40,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:40,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:41,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:42,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:43,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:44,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:45,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:45,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:46,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:47,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:48,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:49,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:50,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:50,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:51,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:52,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:53,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:54,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:55,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:55,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:53:56,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:57,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:58,139 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:53:58,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:53:59,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:00,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:01,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:01,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:02,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:03,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:04,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:05,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:06,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:06,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:07,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:08,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:54:08,078 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:54:08,079 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:54:08,080 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:54:08,087 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:54:08,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:09,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:10,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:11,090 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:54:11,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:12,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:12,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:13,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:14,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:15,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:16,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:17,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:17,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:18,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:19,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:20,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:21,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:22,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:22,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:23,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:24,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:25,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:26,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:27,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:27,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:28,143 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:54:28,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:29,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:30,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:31,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:32,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:32,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:33,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:34,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:35,871 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:36,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:37,156 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:37,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:38,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:39,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:40,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:54:40,814 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:54:40,815 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:54:40,817 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:54:41,114 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:54:41,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:42,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:42,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:43,115 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:54:43,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:44,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:45,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:46,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:47,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:47,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:48,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:49,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:50,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:51,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:52,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:52,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:53,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:54,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:55,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:56,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:57,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:57,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:54:58,148 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:54:58,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:54:59,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:00,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:01,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:02,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:03,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:03,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:04,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:05,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:06,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:07,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:08,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:08,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:09,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:10,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:11,276 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:12,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:12,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:55:12,389 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:55:12,390 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:55:12,395 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:55:13,140 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:55:13,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:13,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:14,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:15,141 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:55:15,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:16,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:17,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:18,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:18,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:19,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:20,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:21,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:22,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:23,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:23,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:24,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:25,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:26,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:27,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:28,151 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:55:28,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:29,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:29,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:30,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:31,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:32,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:33,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:34,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:34,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:35,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:36,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:37,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:38,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:39,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:39,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:40,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:41,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:42,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:43,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:55:43,177 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:55:43,177 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:55:43,182 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:55:43,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:44,170 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:55:44,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:44,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:45,171 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:55:45,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:46,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:47,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:48,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:49,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:49,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:50,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:51,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:52,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:53,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:54,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:55:54,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:55,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:56,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:57,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:58,155 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:55:58,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:55:59,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:00,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:00,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:01,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:02,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:03,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:04,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:05,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:05,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:06,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:07,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:08,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:09,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:10,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:10,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:11,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:12,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:12,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:56:12,762 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:56:12,762 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:56:12,764 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:56:13,199 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:56:13,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:14,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:15,201 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:56:15,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:15,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:16,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:17,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:18,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:19,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:20,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:20,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:21,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:22,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:23,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:24,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:25,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:25,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:26,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:27,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:28,160 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:56:28,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:29,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:30,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:31,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:31,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:32,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:33,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:34,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:35,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:36,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:36,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:37,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:38,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:39,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:40,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:41,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:41,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:42,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:43,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:44,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:45,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:46,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:46,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:47,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:47,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:56:47,370 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:56:47,370 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:56:47,373 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:56:48,232 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:56:48,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:49,232 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:56:49,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:50,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:51,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:51,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:52,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:53,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:54,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:55,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:56,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:56,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:56:57,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:58,163 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:56:58,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:56:59,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:00,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:01,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:02,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:02,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:03,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:04,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:05,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:06,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:07,167 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:07,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:08,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:09,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:10,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:11,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:12,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:12,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:13,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:14,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:15,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:16,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:17,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:17,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:17,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:57:17,611 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:57:17,612 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:57:17,615 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:57:18,258 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:57:18,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:19,259 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:57:19,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:20,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:21,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:22,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:22,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:23,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:24,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:25,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:26,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:27,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:27,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:28,167 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:57:28,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:29,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:30,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:31,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:32,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:33,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:33,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:34,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:35,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:36,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:37,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:38,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:38,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:39,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:40,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:41,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:42,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:43,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:43,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:44,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:45,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:46,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:46,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:57:46,860 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:57:46,860 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:57:46,862 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:57:47,287 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:57:47,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:48,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:48,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:49,288 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:57:49,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:50,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:51,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:52,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:53,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:53,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:54,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:55,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:56,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:57,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:58,172 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:57:58,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:57:59,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:57:59,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:00,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:01,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:02,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:03,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:04,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:04,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:05,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:06,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:07,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:08,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:09,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:09,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:10,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:11,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:12,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:13,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:14,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:14,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:15,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:16,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:17,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:18,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:58:18,320 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:58:18,321 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:58:18,322 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:58:18,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:19,312 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:58:19,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:19,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:20,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:21,314 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:58:21,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:22,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:23,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:24,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:24,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:25,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:26,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:27,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:28,176 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:58:28,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:29,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:30,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:30,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:31,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:32,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:33,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:34,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:35,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:35,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:36,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:37,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:38,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:39,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:40,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:40,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:41,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:42,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:43,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:44,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:45,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:45,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:45,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:58:45,865 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:58:45,865 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:58:45,867 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:58:46,338 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:58:46,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:47,339 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:58:47,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:48,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:49,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:50,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:50,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:51,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:52,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:53,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:54,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:55,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:55,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:58:56,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:57,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:58,180 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:58:58,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:58:59,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:00,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:01,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:01,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:02,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:03,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:04,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:05,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:06,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:06,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:07,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:08,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:09,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:10,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:11,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:11,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:12,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:13,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:14,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:14,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:59:14,739 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:59:14,739 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:59:14,740 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:59:15,361 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:59:15,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:16,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:16,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:17,362 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:59:17,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:18,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:19,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:20,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:21,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:21,781 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:22,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:23,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:24,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:25,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:26,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:26,781 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:27,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:28,184 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:59:28,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:29,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:30,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:31,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:32,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:32,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:33,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:34,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:35,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:36,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:37,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:37,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:38,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:39,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:40,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:41,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:42,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:42,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:42,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 12:59:42,882 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 12:59:42,882 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 12:59:42,884 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 12:59:43,389 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 12:59:43,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:44,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:45,390 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 12:59:45,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:46,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:47,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:47,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:48,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:49,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:50,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:51,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:52,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:52,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:53,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:54,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:55,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:56,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:57,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:57,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 12:59:58,188 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 12:59:58,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 12:59:59,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:00,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:01,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:02,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:03,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:03,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:04,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:05,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:06,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:07,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:08,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:08,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:09,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:10,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:10,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:00:10,752 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:00:10,753 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:00:10,754 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:00:11,413 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:00:11,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:12,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:13,415 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:00:13,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:13,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:14,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:15,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:16,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:17,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:18,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:18,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:19,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:20,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:21,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:22,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:23,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:23,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:24,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:25,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:26,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:27,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:28,193 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:00:28,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:29,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:29,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:30,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:31,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:32,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:33,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:34,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:34,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:35,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:36,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:37,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:38,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:39,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:39,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:40,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:41,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:42,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:43,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:43,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:00:43,937 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:00:43,938 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:00:43,939 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:00:44,447 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:00:44,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:44,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:45,448 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:00:45,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:46,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:47,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:48,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:49,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:49,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:50,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:51,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:52,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:53,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:54,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:54,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:00:55,642 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:56,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:57,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:58,195 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:00:58,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:00:59,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:00,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:00,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:01,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:02,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:03,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:04,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:05,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:05,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:06,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:07,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:08,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:09,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:10,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:10,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:11,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:12,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:01:12,101 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:01:12,101 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:01:12,103 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:01:12,479 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:01:12,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:13,480 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:01:13,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:14,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:15,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:16,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:16,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:17,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:18,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:19,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:20,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:21,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:21,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:22,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:23,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:24,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:25,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:26,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:26,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:27,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:28,199 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:01:28,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:29,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:30,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:31,201 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:31,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:32,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:33,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:34,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:35,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:36,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:36,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:37,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:38,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:39,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:40,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:41,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:41,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:42,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:43,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:43,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:01:43,795 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:01:43,796 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:01:43,798 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:01:44,504 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:01:44,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:45,505 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:01:45,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:46,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:46,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:47,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:48,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:49,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:50,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:51,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:51,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:52,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:53,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:54,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:55,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:56,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:56,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:01:57,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:58,203 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:01:58,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:01:59,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:00,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:01,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:02,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:02,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:03,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:04,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:05,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:06,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:07,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:07,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:08,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:09,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:10,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:11,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:12,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:12,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:12,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:02:12,962 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:02:12,963 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:02:12,965 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:02:13,526 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:02:13,527 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:02:13,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:14,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:15,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:16,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:17,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:18,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:18,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:19,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:20,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:21,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:22,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:23,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:23,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:24,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:25,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:26,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:27,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:28,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:28,207 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:02:28,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:29,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:30,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:31,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:32,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:33,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:33,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:34,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:35,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:36,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:37,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:38,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:38,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:39,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:40,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:41,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:42,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:02:42,091 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:02:42,092 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:02:42,093 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:02:42,552 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:02:42,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:43,552 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:02:43,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:44,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:44,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:45,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:46,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:47,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:48,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:49,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:49,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:50,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:51,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:52,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:53,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:54,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:54,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:55,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:56,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:57,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:58,211 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:02:58,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:02:59,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:02:59,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:00,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:01,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:02,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:03,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:04,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:04,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:05,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:06,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:07,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:08,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:09,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:09,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:10,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:11,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:12,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:03:12,038 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:03:12,038 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:03:12,040 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:03:12,584 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:03:12,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:13,585 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:03:13,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:14,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:15,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:15,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:16,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:17,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:18,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:19,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:20,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:20,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:21,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:22,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:23,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:24,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:25,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:25,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:26,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:27,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:28,215 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:03:28,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:29,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:30,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:30,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:31,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:32,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:33,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:34,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:35,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:35,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:36,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:37,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:38,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:39,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:03:39,669 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:03:39,669 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:03:39,670 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:03:39,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:40,617 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:03:40,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:40,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:41,618 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:03:41,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:42,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:43,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:44,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:45,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:45,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:46,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:47,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:48,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:49,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:50,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:50,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:51,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:52,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:53,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:54,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:55,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:55,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:03:56,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:57,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:58,219 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:03:58,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:03:59,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:00,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:01,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:01,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:02,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:03,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:04,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:05,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:06,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:06,687 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:07,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:08,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:09,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:04:09,276 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:04:09,277 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:04:09,278 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:04:09,653 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:04:09,653 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:04:09,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:10,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:11,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:11,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:12,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:13,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:14,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:15,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:16,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:16,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:17,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:18,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:19,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:20,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:21,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:21,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:22,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:23,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:24,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:25,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:26,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:26,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:27,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:28,224 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:04:28,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:29,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:30,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:31,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:32,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:32,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:33,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:34,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:35,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:36,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:37,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:37,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:38,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:39,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:40,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:04:40,550 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:04:40,551 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:04:40,553 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:04:40,683 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:04:40,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:41,684 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:04:41,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:42,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:42,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:43,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:44,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:45,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:46,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:47,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:47,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:48,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:49,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:50,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:52,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:52,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:53,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:54,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:55,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:56,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:57,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:57,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:04:58,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:04:58,228 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:04:59,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:00,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:01,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:02,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:03,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:03,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:04,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:05,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:06,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:07,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:08,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:08,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:09,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:09,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:05:09,379 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:05:09,380 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:05:09,381 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:05:09,718 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:05:09,718 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:05:10,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:11,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:12,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:13,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:13,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:14,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:15,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:16,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:17,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:18,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:18,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:19,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:20,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:21,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:22,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:23,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:23,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:24,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:25,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:26,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:27,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:28,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:28,232 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:05:29,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:29,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:30,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:31,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:32,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:33,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:34,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:34,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:35,072 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:36,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:37,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:38,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:39,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:39,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:39,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:05:39,895 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:05:39,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:05:39,896 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:05:40,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:40,750 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:05:41,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:41,751 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:05:42,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:43,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:44,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:44,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:45,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:46,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:47,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:48,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:49,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:49,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:50,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:51,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:52,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:53,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:54,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:54,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:05:55,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:56,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:57,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:58,237 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:05:58,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:05:59,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:00,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:00,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:01,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:02,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:03,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:04,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:05,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:05,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:06,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:07,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:08,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:08,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:06:08,691 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:06:08,691 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:06:08,692 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:06:08,773 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:06:09,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:09,774 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:06:10,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:10,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:11,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:12,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:13,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:14,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:15,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:15,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:16,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:17,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:18,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:19,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:20,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:20,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:21,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:22,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:23,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:24,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:25,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:25,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:26,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:27,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:28,240 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:06:28,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:29,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:30,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:31,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:31,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:32,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:33,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:34,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:35,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:36,248 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:36,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:37,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:38,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:39,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:40,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:40,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:06:40,720 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:06:40,720 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:06:40,721 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:06:40,802 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:06:41,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:41,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:41,803 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:06:42,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:43,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:44,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:45,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:46,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:46,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:47,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:48,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:49,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:50,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:51,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:51,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:52,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:53,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:54,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:55,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:56,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:56,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:06:57,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:58,244 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:06:58,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:06:59,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:00,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:01,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:02,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:02,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:03,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:04,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:05,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:06,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:07,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:07,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:08,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:07:08,247 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:07:08,248 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:07:08,249 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:07:08,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:08,826 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:07:09,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:09,827 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:07:10,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:11,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:12,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:12,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:13,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:14,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:15,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:16,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:17,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:17,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:18,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:19,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:20,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:21,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:22,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:22,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:23,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:24,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:25,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:26,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:27,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:27,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:28,248 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:07:28,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:29,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:30,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:31,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:32,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:33,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:33,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:34,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:35,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:36,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:37,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:07:37,225 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:07:37,226 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:07:37,227 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:07:37,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:37,857 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:07:37,858 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:07:38,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:38,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:39,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:40,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:41,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:42,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:43,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:43,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:44,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:45,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:46,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:47,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:48,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:48,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:49,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:50,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:51,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:52,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:53,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:53,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:54,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:55,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:56,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:57,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:58,252 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:07:58,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:07:59,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:07:59,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:00,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:01,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:02,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:03,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:04,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:04,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:05,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:06,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:07,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:08:07,519 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:08:07,519 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:08:07,521 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:08:07,885 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:08:07,886 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:08:07,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:08,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:09,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:09,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:10,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:11,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:12,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:13,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:14,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:14,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:15,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:16,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:17,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:18,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:19,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:19,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:20,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:21,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:22,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:23,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:24,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:24,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:25,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:26,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:27,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:28,257 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:08:28,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:29,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:30,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:30,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:31,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:32,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:33,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:34,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:35,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:35,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:36,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:36,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:08:36,680 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:08:36,681 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:08:36,682 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:08:36,917 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:08:37,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:37,918 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:08:38,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:39,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:40,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:40,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:41,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:42,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:43,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:44,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:45,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:45,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:46,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:47,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:48,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:49,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:50,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:50,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:51,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:52,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:53,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:54,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:55,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:08:55,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:56,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:57,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:58,260 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:08:58,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:08:59,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:00,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:01,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:01,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:02,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:03,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:04,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:05,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:06,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:06,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:07,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:08,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:09,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:10,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:09:10,571 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:09:10,572 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:09:10,573 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:09:10,945 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:09:10,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:11,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:11,946 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:09:11,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:12,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:13,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:14,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:15,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:16,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:16,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:17,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:18,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:19,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:20,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:21,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:21,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:22,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:24,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:25,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:26,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:26,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:27,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:28,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:28,264 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:09:29,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:30,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:31,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:32,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:32,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:33,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:34,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:35,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:36,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:37,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:37,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:38,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:39,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:40,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:41,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:41,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:09:41,313 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:09:41,314 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:09:41,315 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:09:41,970 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:09:41,971 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:09:42,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:42,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:43,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:44,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:45,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:46,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:47,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:47,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:48,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:49,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:50,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:51,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:52,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:52,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:53,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:54,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:55,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:56,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:57,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:57,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:09:58,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:09:58,268 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:09:59,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:00,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:01,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:02,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:03,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:03,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:04,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:05,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:06,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:07,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:08,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:08,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:09,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:10,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:11,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:11,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:10:11,891 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:10:11,891 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:10:11,893 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:10:11,999 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:10:12,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:13,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:13,931 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:14,000 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:10:14,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:15,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:16,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:17,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:18,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:18,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:19,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:20,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:21,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:22,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:23,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:23,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:24,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:25,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:26,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:27,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:28,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:28,272 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:10:29,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:29,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:30,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:31,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:32,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:33,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:34,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:34,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:35,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:36,036 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:37,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:38,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:39,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:39,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:39,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:10:39,444 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:10:39,445 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:10:39,446 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:10:40,025 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:10:40,025 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:10:40,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:41,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:42,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:43,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:44,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:44,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:45,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:46,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:47,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:48,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:49,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:49,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:50,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:51,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:52,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:53,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:54,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:54,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:10:55,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:56,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:57,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:58,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:10:58,276 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:10:59,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:00,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:00,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:01,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:02,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:03,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:04,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:05,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:05,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:06,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:07,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:08,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:09,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:10,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:10,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:11:10,201 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:11:10,201 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:11:10,203 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:11:11,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:11,050 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:11:11,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:12,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:13,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:13,052 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:11:14,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:15,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:16,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:16,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:17,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:18,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:19,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:20,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:21,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:21,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:22,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:23,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:24,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:25,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:26,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:26,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:27,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:28,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:28,279 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:11:29,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:30,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:31,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:31,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:32,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:33,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:34,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:35,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:36,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:36,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:37,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:38,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:39,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:40,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:41,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:11:41,028 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:11:41,029 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:11:41,030 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:11:41,078 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:11:41,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:42,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:42,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:43,080 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:11:43,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:44,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:45,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:46,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:47,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:47,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:48,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:49,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:50,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:51,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:52,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:52,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:53,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:54,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:55,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:56,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:57,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:11:57,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:58,283 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:11:58,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:11:59,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:00,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:01,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:02,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:02,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:03,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:04,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:05,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:06,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:07,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:07,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:08,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:09,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:10,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:11,156 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:12:11,157 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:12:11,158 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:12:11,160 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:12:11,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:12,111 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:12:12,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:13,112 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:12:13,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:13,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:14,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:15,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:16,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:17,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:18,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:18,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:19,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:20,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:21,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:22,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:23,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:23,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:24,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:25,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:26,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:27,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:28,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:28,287 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:12:28,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:29,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:30,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:31,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:32,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:33,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:33,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:34,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:35,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:36,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:37,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:38,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:38,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:39,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:40,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:40,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:12:40,750 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:12:40,751 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:12:40,752 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:12:41,142 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:12:41,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:42,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:43,144 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:12:43,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:43,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:44,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:45,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:46,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:47,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:48,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:48,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:49,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:50,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:51,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:52,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:53,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:53,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:54,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:55,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:56,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:57,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:58,291 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:12:58,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:12:59,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:12:59,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:00,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:01,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:02,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:03,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:04,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:04,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:05,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:06,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:07,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:08,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:09,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:09,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:10,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:11,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:13:11,213 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:13:11,214 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:13:11,215 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:13:11,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:12,167 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:13:12,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:13,168 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:13:13,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:14,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:15,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:15,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:16,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:17,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:18,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:19,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:20,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:20,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:21,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:22,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:23,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:24,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:25,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:25,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:26,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:27,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:28,295 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:13:28,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:29,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:30,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:30,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:31,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:32,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:33,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:34,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:35,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:35,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:36,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:37,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:38,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:39,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:40,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:40,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:41,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:42,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:42,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:13:42,784 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:13:42,785 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:13:42,786 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:13:43,197 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:13:43,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:44,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:45,198 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:13:45,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:45,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:46,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:47,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:48,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:49,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:50,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:50,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:51,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:52,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:53,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:54,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:55,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:55,825 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:13:56,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:57,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:58,300 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:13:58,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:13:59,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:00,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:01,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:01,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:02,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:03,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:04,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:05,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:06,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:06,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:07,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:08,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:09,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:10,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:11,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:11,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:12,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:13,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:14,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:15,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:16,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:16,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:14:16,318 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:14:16,318 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:14:16,320 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:14:16,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:17,225 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:14:17,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:18,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:19,226 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:14:19,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:20,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:21,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:21,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:22,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:23,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:24,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:25,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:26,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:26,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:27,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:28,303 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:14:28,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:29,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:30,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:31,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:32,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:32,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:33,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:34,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:35,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:36,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:37,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:37,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:38,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:39,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:40,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:41,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:42,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:42,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:43,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:44,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:45,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:46,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:47,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:14:47,309 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:14:47,309 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:14:47,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:47,311 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:14:47,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:48,249 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:14:48,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:49,250 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:14:49,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:50,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:51,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:52,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:52,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:53,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:54,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:55,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:56,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:57,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:14:57,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:58,307 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:14:58,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:14:59,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:00,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:01,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:02,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:03,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:03,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:04,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:05,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:06,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:07,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:08,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:08,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:09,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:10,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:11,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:12,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:13,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:13,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:14,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:15,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:16,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:16,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:15:16,895 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:15:16,895 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:15:16,896 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:15:17,280 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:15:17,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:18,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:18,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:19,281 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:15:19,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:20,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:21,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:22,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:23,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:23,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:24,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:25,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:26,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:27,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:28,311 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:15:28,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:29,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:29,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:30,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:32,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:32,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:33,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:34,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:34,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:35,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:36,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:37,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:38,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:39,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:39,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:40,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:41,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:42,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:43,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:44,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:44,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:45,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:46,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:47,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:48,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:49,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:49,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:50,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:15:50,482 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:15:50,483 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:15:50,484 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:15:50,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:51,309 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:15:51,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:52,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:53,310 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:15:53,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:54,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:15:54,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:55,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:56,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:57,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:58,315 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:15:58,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:15:59,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:00,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:00,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:01,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:02,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:03,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:04,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:05,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:05,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:06,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:07,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:08,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:09,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:10,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:10,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:11,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:12,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:13,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:14,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:15,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:15,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:16,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:17,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:18,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:19,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:20,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:20,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:21,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:22,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:16:22,264 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:16:22,264 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:16:22,266 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:16:22,349 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:16:22,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:23,350 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:16:23,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:24,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:25,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:26,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:26,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:27,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:28,319 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:16:28,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:29,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:30,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:31,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:31,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:32,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:33,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:34,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:35,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:36,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:37,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:37,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:38,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:39,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:40,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:41,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:41,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:42,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:43,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:44,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:45,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:46,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:46,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:47,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:49,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:50,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:51,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:51,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:52,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:53,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:54,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:54,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:16:54,634 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:16:54,635 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:16:54,636 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:16:55,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:55,380 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:16:56,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:56,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:16:57,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:57,382 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:16:58,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:16:58,323 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:16:59,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:00,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:01,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:02,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:02,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:03,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:04,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:05,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:06,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:07,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:07,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:08,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:09,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:10,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:11,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:12,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:12,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:13,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:14,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:15,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:16,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:17,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:17,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:18,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:19,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:20,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:21,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:22,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:22,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:23,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:24,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:25,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:25,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:17:25,092 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:17:25,092 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:17:25,094 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:17:25,405 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:17:26,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:27,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:27,407 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:17:28,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:28,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:28,327 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:17:29,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:30,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:31,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:32,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:33,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:33,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:34,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:35,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:36,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:37,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:38,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:38,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:39,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:40,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:41,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:42,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:43,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:43,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:44,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:45,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:46,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:47,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:48,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:48,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:49,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:50,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:51,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:52,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:53,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:53,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:17:53,315 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:17:53,315 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:17:53,316 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:17:53,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:17:53,432 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:17:54,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:55,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:55,434 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:17:56,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:57,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:58,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:58,331 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:17:59,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:17:59,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:00,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:01,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:02,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:03,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:04,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:04,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:05,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:06,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:07,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:08,169 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:09,169 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:09,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:10,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:11,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:12,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:13,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:14,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:14,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:15,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:16,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:17,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:18,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:19,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:19,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:20,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:21,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:22,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:23,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:24,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:24,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:25,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:26,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:26,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:18:26,480 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:18:26,481 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:18:26,482 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:18:27,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:27,456 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:18:27,457 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:18:28,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:28,335 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:18:29,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:30,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:30,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:31,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:32,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:33,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:34,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:35,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:35,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:36,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:37,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:38,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:39,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:40,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:40,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:41,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:42,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:43,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:44,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:45,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:45,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:46,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:47,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:48,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:49,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:50,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:50,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:51,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:52,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:53,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:54,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:55,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:55,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:18:56,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:56,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:18:56,894 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:18:56,894 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:18:56,895 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:18:57,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:57,483 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:18:58,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:58,339 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:18:59,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:18:59,485 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:19:00,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:01,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:01,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:02,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:03,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:04,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:05,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:06,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:06,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:07,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:08,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:09,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:10,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:11,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:11,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:12,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:13,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:14,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:15,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:16,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:16,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:17,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:18,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:19,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:20,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:21,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:21,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:22,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:23,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:24,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:25,318 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:26,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:26,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:27,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:28,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:28,348 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:19:29,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:29,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:19:29,453 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:19:29,454 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:19:29,455 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:19:29,506 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:19:30,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:31,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:31,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:31,508 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:19:32,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:33,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:34,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:35,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:36,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:36,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:37,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:38,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:39,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:40,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:41,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:41,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:42,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:43,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:44,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:45,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:46,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:46,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:47,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:48,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:49,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:50,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:51,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:51,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:52,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:53,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:54,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:55,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:56,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:56,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:19:57,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:19:57,141 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:19:57,141 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:19:57,142 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:19:57,533 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:19:57,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:58,349 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:19:58,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:19:59,534 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:19:59,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:00,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:01,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:02,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:02,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:03,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:04,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:05,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:06,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:07,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:07,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:08,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:09,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:10,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:11,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:12,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:12,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:13,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:14,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:15,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:16,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:17,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:17,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:18,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:19,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:20,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:21,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:22,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:22,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:23,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:24,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:25,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:26,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:27,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:27,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:28,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:20:28,262 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:20:28,263 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:20:28,264 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:20:28,353 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:20:28,562 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:20:28,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:29,562 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:20:29,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:30,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:31,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:32,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:33,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:33,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:34,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:35,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:36,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:37,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:38,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:38,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:39,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:40,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:41,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:42,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:43,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:43,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:44,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:45,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:46,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:47,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:48,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:48,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:49,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:50,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:51,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:52,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:53,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:53,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:54,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:55,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:56,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:57,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:20:57,518 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:20:57,518 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:20:57,519 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:20:57,602 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:20:57,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:58,357 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:20:58,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:20:59,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:20:59,603 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:20:59,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:00,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:01,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:02,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:03,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:04,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:04,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:05,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:06,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:07,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:08,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:09,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:09,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:10,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:11,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:12,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:13,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:14,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:14,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:15,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:16,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:17,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:18,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:19,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:19,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:20,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:21,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:22,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:23,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:24,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:24,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:25,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:26,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:27,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:21:27,715 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:21:27,715 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:21:27,717 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:21:28,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:28,361 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:21:28,644 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:21:29,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:29,645 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:21:30,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:30,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:31,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:32,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:33,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:34,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:35,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:35,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:36,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:37,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:38,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:39,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:40,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:40,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:41,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:42,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:43,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:44,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:45,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:45,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:46,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:47,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:48,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:49,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:50,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:50,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:51,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:52,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:53,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:54,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:55,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:55,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:21:56,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:57,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:57,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:21:57,848 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:21:57,848 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:21:57,850 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:21:58,283 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:58,364 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:21:58,670 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:21:59,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:21:59,671 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:22:00,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:01,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:01,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:02,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:03,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:04,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:05,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:06,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:06,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:07,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:08,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:09,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:10,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:11,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:11,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:12,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:13,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:14,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:15,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:16,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:16,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:17,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:18,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:19,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:20,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:21,290 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:21,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:22,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:23,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:24,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:25,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:26,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:22:26,275 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:22:26,275 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:22:26,277 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:22:26,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:26,697 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:22:27,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:27,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:27,698 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:22:28,368 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:22:28,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:29,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:30,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:31,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:32,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:32,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:33,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:34,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:35,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:36,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:37,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:37,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:38,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:39,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:40,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:41,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:42,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:42,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:43,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:44,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:45,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:46,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:47,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:47,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:48,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:49,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:50,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:51,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:52,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:52,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:53,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:54,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:55,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:56,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:57,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:22:57,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:58,369 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:22:58,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:58,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:22:58,624 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:22:58,625 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:22:58,626 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:22:58,729 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:22:59,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:22:59,730 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:23:00,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:01,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:02,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:02,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:03,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:04,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:05,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:06,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:07,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:07,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:08,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:09,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:10,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:11,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:12,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:12,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:13,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:14,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:15,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:16,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:17,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:17,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:18,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:19,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:20,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:21,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:22,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:22,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:23,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:24,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:25,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:26,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:27,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:27,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:28,374 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:23:28,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:29,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:30,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:31,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:23:31,316 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:23:31,316 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:23:31,318 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:23:31,765 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:23:32,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:33,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:33,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:33,766 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:23:34,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:35,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:36,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:37,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:38,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:38,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:39,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:40,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:41,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:42,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:43,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:43,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:44,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:45,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:46,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:47,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:48,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:48,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:49,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:50,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:51,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:52,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:53,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:53,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:54,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:55,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:56,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:57,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:58,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:23:58,378 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:23:58,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:23:59,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:00,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:01,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:02,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:03,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:03,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:03,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:24:03,612 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:24:03,612 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:24:03,614 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:24:03,792 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:24:04,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:05,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:05,794 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:24:06,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:07,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:08,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:08,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:09,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:10,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:11,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:12,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:13,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:13,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:14,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:15,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:16,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:17,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:18,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:18,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:19,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:20,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:21,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:22,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:23,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:23,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:24,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:25,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:26,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:27,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:28,381 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:24:28,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:29,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:29,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:30,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:31,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:32,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:33,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:34,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:34,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:35,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:36,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:37,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:37,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:24:37,940 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:24:37,940 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:24:37,942 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:24:38,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:38,825 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:24:39,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:39,826 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:24:39,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:40,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:41,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:42,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:43,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:44,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:44,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:45,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:46,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:47,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:48,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:49,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:49,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:50,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:51,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:52,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:53,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:54,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:54,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:24:55,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:56,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:57,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:58,386 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:24:58,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:24:59,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:00,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:00,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:01,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:02,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:03,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:04,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:05,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:25:05,201 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:25:05,202 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:25:05,203 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:25:05,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:05,846 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:25:06,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:06,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:07,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:07,848 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:25:08,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:09,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:10,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:11,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:11,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:12,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:13,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:14,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:15,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:16,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:16,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:17,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:18,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:19,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:20,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:21,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:21,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:22,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:23,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:24,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:25,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:26,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:26,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:27,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:28,389 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:25:28,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:29,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:30,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:31,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:31,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:32,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:33,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:34,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:35,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:36,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:36,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:37,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:25:37,363 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:25:37,363 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:25:37,365 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:25:37,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:37,879 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:25:38,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:39,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:39,881 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:25:40,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:41,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:41,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:42,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:43,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:44,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:45,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:46,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:46,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:47,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:48,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:49,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:50,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:51,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:51,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:52,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:53,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:54,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:55,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:56,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:25:56,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:57,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:58,393 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:25:58,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:25:59,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:00,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:01,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:02,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:02,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:03,781 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:04,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:05,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:06,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:26:06,504 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:26:06,505 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:26:06,506 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:26:06,908 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:26:06,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:07,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:07,909 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:26:07,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:08,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:09,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:10,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:11,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:12,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:12,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:13,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:14,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:15,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:16,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:17,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:17,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:18,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:19,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:20,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:21,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:22,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:22,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:23,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:24,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:25,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:26,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:27,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:27,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:28,397 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:26:28,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:29,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:30,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:31,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:32,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:33,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:33,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:34,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:35,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:36,642 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:26:36,643 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:26:36,644 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:26:36,645 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:26:36,934 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:26:37,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:37,935 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:26:38,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:38,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:39,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:40,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:41,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:42,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:43,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:43,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:44,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:45,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:46,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:47,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:48,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:48,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:49,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:50,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:51,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:52,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:53,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:53,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:26:54,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:55,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:56,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:57,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:58,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:58,401 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:26:59,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:26:59,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:00,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:01,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:02,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:03,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:04,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:04,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:05,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:06,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:07,055 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:08,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:09,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:09,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:09,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:27:09,575 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:27:09,575 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:27:09,576 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:27:09,965 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:27:10,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:11,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:11,967 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:27:12,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:13,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:14,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:14,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:15,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:16,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:17,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:18,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:19,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:19,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:20,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:21,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:22,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:23,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:24,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:24,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:25,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:26,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:27,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:28,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:28,406 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:27:29,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:30,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:30,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:31,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:32,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:33,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:34,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:35,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:35,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:36,053 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:37,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:38,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:39,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:40,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:40,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:41,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:42,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:42,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:27:42,805 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:27:42,806 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:27:42,806 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:27:42,994 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:27:43,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:43,995 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:27:44,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:45,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:45,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:46,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:47,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:48,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:49,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:50,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:50,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:51,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:52,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:53,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:54,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:55,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:55,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:27:56,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:57,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:58,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:27:58,409 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:27:59,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:00,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:01,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:01,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:02,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:03,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:04,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:05,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:06,250 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:06,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:07,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:08,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:09,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:10,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:11,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:11,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:12,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:12,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:28:12,541 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:28:12,541 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:28:12,543 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:28:13,025 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:28:13,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:14,027 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:28:14,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:15,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:16,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:16,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:17,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:18,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:19,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:20,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:21,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:21,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:22,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:23,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:24,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:25,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:26,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:26,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:27,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:28,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:28,413 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:28:29,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:30,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:31,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:32,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:32,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:33,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:34,257 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:35,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:36,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:37,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:37,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:38,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:39,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:40,266 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:40,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:28:40,908 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:28:40,909 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:28:40,910 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:28:41,050 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:28:41,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:42,050 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:28:42,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:42,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:43,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:44,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:45,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:46,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:47,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:47,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:48,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:49,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:50,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:51,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:52,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:52,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:53,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:54,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:55,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:56,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:57,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:57,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:28:58,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:28:58,417 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:28:59,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:00,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:01,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:02,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:03,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:03,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:04,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:05,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:06,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:07,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:08,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:08,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:09,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:10,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:29:10,186 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:29:10,187 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:29:10,188 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:29:10,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:11,078 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:29:11,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:12,079 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:29:12,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:13,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:14,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:14,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:15,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:16,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:17,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:18,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:19,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:19,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:20,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:21,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:22,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:23,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:24,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:24,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:25,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:26,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:27,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:28,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:28,421 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:29:29,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:29,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:30,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:31,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:32,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:33,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:34,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:34,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:35,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:36,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:37,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:38,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:39,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:39,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:40,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:41,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:42,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:43,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:43,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:29:43,367 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:29:43,368 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:29:43,369 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:29:44,112 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:29:44,112 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:29:44,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:45,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:45,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:46,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:47,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:48,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:49,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:50,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:50,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:51,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:52,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:53,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:54,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:55,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:55,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:29:56,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:57,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:58,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:29:58,425 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:29:59,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:00,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:00,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:01,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:02,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:03,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:04,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:05,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:05,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:06,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:07,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:08,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:09,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:10,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:10,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:11,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:12,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:13,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:13,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:30:13,993 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:30:13,994 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:30:13,996 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:30:14,142 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:30:14,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:15,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:16,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:16,143 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:30:16,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:17,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:18,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:19,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:20,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:21,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:21,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:22,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:23,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:24,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:25,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:26,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:26,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:27,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:28,429 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:30:28,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:29,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:30,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:31,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:31,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:32,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:33,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:34,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:35,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:36,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:36,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:37,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:38,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:39,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:40,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:41,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:41,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:42,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:43,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:44,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:45,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:30:45,085 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:30:45,086 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:30:45,087 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:30:45,170 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:30:45,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:46,171 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:30:46,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:47,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:47,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:48,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:49,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:50,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:51,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:52,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:52,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:53,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:54,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:55,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:56,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:57,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:30:57,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:58,433 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:30:58,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:30:59,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:00,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:01,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:02,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:02,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:03,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:04,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:05,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:06,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:07,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:07,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:08,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:09,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:10,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:11,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:12,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:12,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:13,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:14,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:15,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:31:15,256 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:31:15,256 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:31:15,257 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:31:15,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:16,194 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:31:16,195 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:31:16,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:17,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:18,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:18,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:19,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:20,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:21,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:22,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:23,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:23,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:24,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:25,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:26,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:27,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:28,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:28,437 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:31:28,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:29,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:30,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:31,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:32,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:33,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:33,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:34,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:35,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:36,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:37,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:38,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:38,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:39,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:40,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:41,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:42,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:43,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:43,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:44,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:31:44,660 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:31:44,661 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:31:44,662 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:31:45,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:45,218 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:31:45,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:46,219 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:31:46,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:47,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:48,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:48,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:49,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:50,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:51,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:52,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:53,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:53,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:54,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:55,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:56,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:57,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:58,441 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:31:58,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:31:59,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:31:59,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:00,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:01,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:02,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:03,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:04,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:04,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:05,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:06,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:07,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:08,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:09,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:09,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:10,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:11,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:12,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:13,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:14,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:14,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:15,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:32:15,012 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:32:15,012 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:32:15,013 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:32:15,241 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:32:15,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:16,242 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:32:16,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:17,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:18,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:19,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:20,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:20,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:21,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:22,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:23,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:24,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:25,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:25,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:26,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:27,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:28,445 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:32:28,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:29,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:30,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:30,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:31,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:32,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:33,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:34,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:35,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:35,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:36,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:37,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:38,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:39,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:40,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:40,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:41,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:42,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:42,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:32:42,772 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:32:42,773 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:32:42,774 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:32:43,271 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:32:43,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:44,272 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:32:44,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:45,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:45,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:46,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:47,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:48,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:49,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:50,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:50,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:51,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:52,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:53,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:54,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:55,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:55,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:32:56,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:57,697 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:58,448 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:32:58,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:32:59,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:00,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:01,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:01,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:02,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:03,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:04,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:05,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:06,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:06,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:07,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:08,702 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:09,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:10,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:11,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:11,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:12,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:13,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:14,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:33:14,257 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:33:14,258 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:33:14,259 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:33:14,306 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:33:14,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:15,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:16,307 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:33:16,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:17,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:17,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:18,704 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:19,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:20,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:21,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:22,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:22,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:23,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:24,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:25,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:26,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:27,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:27,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:28,451 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:33:28,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:29,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:30,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:31,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:32,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:32,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:33,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:34,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:35,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:36,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:37,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:37,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:38,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:39,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:40,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:41,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:42,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:42,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:43,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:44,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:33:44,573 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:33:44,573 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:33:44,575 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:33:44,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:45,332 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:33:45,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:46,333 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:33:46,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:47,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:47,720 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:48,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:49,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:50,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:51,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:52,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:52,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:53,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:54,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:55,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:56,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:57,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:33:57,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:58,455 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:33:58,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:33:59,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:00,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:01,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:02,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:03,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:03,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:04,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:05,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:06,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:07,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:08,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:08,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:09,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:10,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:11,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:12,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:13,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:13,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:14,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:14,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:34:14,824 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:34:14,825 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:34:14,826 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:34:15,355 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:34:15,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:16,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:17,357 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:34:17,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:18,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:18,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:19,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:20,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:21,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:22,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:23,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:23,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:24,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:25,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:26,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:27,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:28,460 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:34:28,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:29,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:29,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:30,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:31,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:32,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:33,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:34,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:34,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:35,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:36,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:37,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:38,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:39,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:39,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:40,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:41,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:42,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:43,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:44,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:44,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:44,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:34:44,910 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:34:44,911 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:34:44,912 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:34:45,384 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:34:45,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:46,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:47,386 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:34:47,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:48,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:49,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:49,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:50,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:51,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:52,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:53,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:54,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:54,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:34:55,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:56,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:57,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:58,463 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:34:58,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:34:59,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:00,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:00,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:01,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:02,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:03,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:04,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:05,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:05,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:06,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:07,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:08,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:09,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:10,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:10,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:11,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:12,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:13,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:14,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:15,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:35:15,206 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:35:15,206 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:35:15,207 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:35:15,414 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:35:15,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:16,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:16,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:17,416 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:35:17,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:18,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:19,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:20,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:21,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:21,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:22,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:23,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:24,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:25,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:26,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:26,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:27,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:28,468 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:35:28,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:29,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:30,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:31,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:31,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:32,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:33,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:34,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:35,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:36,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:36,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:37,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:38,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:39,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:40,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:41,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:41,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:42,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:43,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:44,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:45,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:35:45,274 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:35:45,275 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:35:45,276 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:35:45,444 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:35:45,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:46,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:47,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:47,446 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:35:47,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:48,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:49,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:50,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:51,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:52,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:52,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:53,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:54,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:55,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:56,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:57,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:35:57,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:58,471 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:35:58,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:35:59,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:00,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:01,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:02,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:02,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:03,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:04,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:05,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:06,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:07,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:07,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:08,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:09,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:10,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:11,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:12,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:12,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:13,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:14,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:15,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:36:15,443 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:36:15,444 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:36:15,445 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:36:15,472 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:36:15,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:16,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:17,473 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:36:17,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:17,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:18,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:19,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:20,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:21,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:22,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:22,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:23,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:24,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:25,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:26,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:27,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:27,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:28,476 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:36:28,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:29,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:30,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:31,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:32,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:33,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:33,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:34,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:35,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:36,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:37,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:38,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:38,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:39,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:40,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:41,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:42,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:43,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:43,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:44,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:45,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:36:45,810 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:36:45,810 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:36:45,812 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:36:46,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:46,508 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:36:47,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:47,509 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:36:48,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:48,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:49,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:50,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:51,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:52,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:53,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:53,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:36:54,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:55,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:56,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:57,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:58,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:58,480 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:36:59,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:36:59,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:00,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:01,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:02,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:03,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:04,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:04,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:05,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:06,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:07,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:08,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:09,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:09,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:10,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:11,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:12,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:13,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:14,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:14,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:15,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:16,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:37:16,234 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:37:16,234 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:37:16,235 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:37:16,530 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:37:16,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:17,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:17,531 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:37:18,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:19,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:20,274 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:20,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:21,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:22,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:23,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:24,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:25,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:25,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:26,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:27,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:28,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:28,483 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:37:29,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:30,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:30,485 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:31,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:32,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:33,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:34,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:35,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:35,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:36,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:37,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:38,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:39,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:40,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:40,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:41,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:42,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:43,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:44,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:45,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:45,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:46,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:46,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:37:46,446 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:37:46,446 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:37:46,447 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:37:46,558 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:37:47,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:47,559 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:37:48,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:49,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:50,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:51,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:51,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:52,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:53,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:54,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:55,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:56,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:56,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:37:57,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:58,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:37:58,487 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:37:59,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:00,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:01,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:01,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:02,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:03,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:04,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:05,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:06,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:06,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:07,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:08,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:09,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:10,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:11,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:11,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:12,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:13,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:14,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:15,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:16,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:38:16,219 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:38:16,219 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:38:16,220 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:38:16,586 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:38:16,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:17,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:17,587 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:38:17,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:18,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:19,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:20,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:21,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:22,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:22,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:23,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:24,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:25,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:26,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:27,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:27,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:28,492 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:38:28,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:29,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:30,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:31,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:32,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:32,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:33,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:34,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:35,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:36,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:37,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:37,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:38,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:39,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:40,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:41,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:42,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:42,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:43,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:44,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:45,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:46,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:38:46,145 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:38:46,145 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:38:46,147 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:38:46,619 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:38:46,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:47,620 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:38:47,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:48,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:48,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:49,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:50,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:51,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:52,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:53,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:53,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:54,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:55,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:56,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:57,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:58,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:38:58,495 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:38:58,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:38:59,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:00,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:01,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:02,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:03,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:03,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:04,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:05,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:06,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:07,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:08,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:08,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:09,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:10,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:11,642 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:12,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:13,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:13,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:14,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:15,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:16,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:39:16,124 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:39:16,124 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:39:16,125 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:39:16,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:16,650 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:39:17,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:17,650 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:39:18,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:19,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:19,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:20,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:21,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:22,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:23,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:24,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:24,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:25,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:26,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:27,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:28,499 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:39:28,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:29,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:29,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:30,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:31,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:32,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:33,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:34,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:34,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:35,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:36,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:37,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:38,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:39,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:39,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:40,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:41,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:42,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:43,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:44,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:44,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:45,654 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:46,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:39:46,497 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:39:46,498 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:39:46,500 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:39:46,682 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:39:46,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:47,683 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:39:47,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:48,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:49,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:49,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:50,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:51,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:52,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:53,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:54,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:39:54,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:55,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:56,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:57,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:58,503 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:39:58,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:39:59,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:00,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:00,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:01,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:02,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:03,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:04,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:05,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:05,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:06,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:07,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:08,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:09,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:10,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:10,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:11,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:12,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:13,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:14,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:15,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:15,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:16,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:40:16,363 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:40:16,364 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:40:16,365 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:40:16,715 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:40:16,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:17,716 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:40:17,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:18,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:19,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:20,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:21,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:21,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:22,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:23,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:24,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:25,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:26,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:26,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:27,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:28,507 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:40:28,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:30,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:31,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:31,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:32,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:33,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:34,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:35,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:36,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:36,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:37,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:38,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:39,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:40,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:41,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:41,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:42,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:43,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:44,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:45,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:46,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:46,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:47,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:47,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:40:47,939 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:40:47,939 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:40:47,940 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:40:48,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:48,745 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:40:49,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:49,746 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:40:50,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:51,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:51,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:52,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:53,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:54,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:55,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:56,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:56,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:40:57,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:58,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:40:58,512 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:40:59,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:00,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:01,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:02,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:02,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:03,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:04,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:05,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:06,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:07,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:07,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:08,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:09,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:10,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:11,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:12,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:12,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:13,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:14,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:15,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:16,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:17,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:17,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:17,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:41:17,907 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:41:17,907 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:41:17,908 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:41:18,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:18,775 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:41:19,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:19,776 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:41:20,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:21,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:22,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:22,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:23,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:24,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:25,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:26,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:27,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:27,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:28,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:28,515 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:41:29,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:30,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:31,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:32,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:33,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:33,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:34,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:35,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:36,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:37,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:38,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:38,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:39,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:40,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:41,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:42,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:43,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:43,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:44,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:45,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:46,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:47,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:41:47,136 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:41:47,137 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:41:47,138 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:41:47,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:47,799 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:41:48,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:49,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:49,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:49,800 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:41:50,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:51,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:52,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:53,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:54,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:54,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:55,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:56,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:57,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:58,519 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:41:58,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:41:59,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:41:59,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:00,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:01,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:02,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:03,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:04,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:04,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:05,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:06,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:07,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:08,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:09,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:09,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:10,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:11,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:12,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:13,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:14,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:14,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:15,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:16,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:17,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:18,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:19,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:19,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:19,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:42:19,772 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:42:19,772 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:42:19,773 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:42:19,828 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:42:20,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:21,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:21,830 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:42:22,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:23,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:24,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:24,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:25,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:26,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:27,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:28,523 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:42:28,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:29,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:30,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:30,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:31,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:32,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:33,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:34,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:35,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:35,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:36,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:37,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:38,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:39,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:40,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:40,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:41,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:42,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:43,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:44,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:45,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:45,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:46,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:47,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:48,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:49,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:50,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:50,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:51,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:52,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:42:52,315 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:42:52,315 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:42:52,317 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:42:52,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:52,855 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:42:53,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:53,856 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:42:54,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:55,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:56,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:42:56,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:57,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:58,527 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:42:58,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:42:59,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:00,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:01,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:01,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:02,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:03,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:04,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:05,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:06,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:06,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:07,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:08,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:09,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:10,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:11,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:11,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:12,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:13,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:14,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:15,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:16,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:16,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:17,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:18,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:19,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:43:19,618 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:43:19,619 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:43:19,620 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:43:19,884 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:43:20,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:21,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:21,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:21,885 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:43:22,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:23,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:24,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:25,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:26,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:26,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:27,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:28,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:28,532 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:43:29,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:30,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:31,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:32,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:32,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:33,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:34,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:35,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:36,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:37,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:37,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:38,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:39,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:40,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:41,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:42,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:42,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:43,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:44,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:45,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:46,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:47,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:47,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:48,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:49,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:50,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:50,256 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:43:50,257 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:43:50,258 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:43:50,259 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:43:50,910 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:43:51,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:51,911 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:43:52,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:53,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:53,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:54,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:55,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:56,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:57,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:58,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:43:58,298 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:43:58,534 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:43:59,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:00,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:01,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:02,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:03,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:03,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:04,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:05,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:06,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:07,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:08,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:08,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:09,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:10,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:11,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:12,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:13,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:13,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:14,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:15,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:16,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:17,169 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:18,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:18,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:19,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:20,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:21,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:21,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:44:21,250 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:44:21,250 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:44:21,252 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:44:21,931 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:44:22,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:23,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:23,935 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:44:24,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:24,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:25,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:26,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:27,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:28,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:28,538 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:44:29,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:29,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:30,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:31,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:32,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:33,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:34,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:34,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:35,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:36,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:37,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:38,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:39,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:39,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:40,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:41,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:42,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:43,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:44,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:44,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:45,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:46,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:47,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:48,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:49,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:49,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:50,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:44:50,013 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:44:50,013 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:44:50,015 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:44:50,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:50,962 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:44:51,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:51,963 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:44:52,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:53,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:54,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:55,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:44:55,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:56,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:57,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:58,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:44:58,542 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:44:59,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:00,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:00,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:01,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:02,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:03,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:04,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:05,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:05,545 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:06,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:07,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:08,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:09,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:10,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:10,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:11,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:12,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:13,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:14,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:15,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:15,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:16,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:17,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:18,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:19,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:20,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:20,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:20,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:45:20,557 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:45:20,558 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:45:20,559 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:45:20,996 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:45:21,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:21,997 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:45:22,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:23,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:24,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:25,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:25,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:26,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:27,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:28,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:28,546 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:45:29,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:30,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:31,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:31,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:32,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:33,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:34,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:35,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:36,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:36,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:37,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:38,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:39,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:40,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:41,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:41,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:42,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:43,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:44,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:45,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:46,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:46,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:47,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:48,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:49,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:50,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:51,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:51,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:52,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:45:52,312 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:45:52,313 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:45:52,314 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:45:52,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:53,028 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:45:53,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:54,029 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:45:54,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:55,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:56,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:57,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:45:57,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:58,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:45:58,550 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:45:59,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:00,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:01,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:02,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:02,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:03,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:04,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:05,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:06,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:07,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:07,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:08,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:09,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:10,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:11,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:12,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:12,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:13,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:14,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:15,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:16,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:17,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:17,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:18,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:19,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:20,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:21,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:22,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:22,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:22,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:46:22,595 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:46:22,596 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:46:22,597 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:46:23,068 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:46:23,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:24,068 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:46:24,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:25,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:26,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:27,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:27,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:28,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:28,555 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:46:29,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:30,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:31,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:32,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:33,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:33,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:34,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:35,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:36,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:37,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:38,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:38,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:39,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:40,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:41,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:42,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:43,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:43,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:44,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:45,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:46,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:47,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:48,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:48,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:49,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:50,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:51,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:52,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:52,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:46:52,736 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:46:52,737 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:46:52,738 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:46:53,100 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:46:53,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:53,776 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:46:54,101 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:46:54,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:55,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:56,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:57,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:58,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:58,558 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:46:59,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:46:59,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:00,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:01,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:02,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:03,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:04,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:04,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:05,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:06,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:07,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:08,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:09,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:09,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:10,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:11,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:12,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:13,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:14,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:14,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:15,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:16,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:17,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:18,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:19,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:19,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:20,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:21,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:22,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:22,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:47:22,955 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:47:22,955 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:47:22,956 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:47:23,134 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:47:23,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:24,135 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:47:24,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:24,995 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:25,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:26,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:27,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:28,562 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:47:28,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:29,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:30,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:30,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:31,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:32,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:33,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:34,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:35,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:35,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:36,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:37,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:38,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:39,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:40,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:40,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:41,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:42,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:43,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:44,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:45,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:45,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:46,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:47,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:48,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:49,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:50,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:50,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:51,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:52,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:53,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:54,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:47:54,311 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:47:54,312 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:47:54,313 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:47:54,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:55,165 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:47:55,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:56,166 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:47:56,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:47:56,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:57,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:58,566 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:47:58,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:47:59,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:00,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:01,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:01,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:02,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:03,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:04,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:05,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:06,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:06,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:07,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:08,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:09,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:10,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:11,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:11,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:12,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:13,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:14,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:15,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:16,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:16,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:17,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:18,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:19,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:20,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:21,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:21,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:22,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:22,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:48:22,923 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:48:22,924 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:48:22,926 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:48:23,193 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:48:23,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:24,194 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:48:24,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:25,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:26,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:26,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:27,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:28,570 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:48:28,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:29,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:30,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:31,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:32,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:32,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:33,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:34,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:35,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:36,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:37,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:37,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:38,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:39,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:40,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:41,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:42,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:42,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:43,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:44,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:45,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:46,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:47,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:47,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:48,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:49,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:50,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:51,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:52,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:52,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:53,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:54,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:48:54,472 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:48:54,473 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:48:54,475 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:48:54,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:55,219 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:48:55,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:56,221 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:48:56,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:57,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:58,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:48:58,573 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:48:58,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:48:59,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:00,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:01,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:02,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:03,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:03,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:04,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:05,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:06,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:07,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:08,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:08,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:09,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:10,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:11,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:12,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:13,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:13,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:14,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:15,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:16,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:17,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:18,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:18,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:19,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:20,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:21,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:22,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:23,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:23,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:24,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:25,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:26,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:27,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:49:27,680 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:49:27,681 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:49:27,682 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:49:28,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:28,252 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:49:28,577 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:49:29,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:29,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:30,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:30,253 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:49:31,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:32,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:33,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:34,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:34,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:35,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:36,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:37,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:38,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:39,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:39,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:40,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:41,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:42,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:43,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:44,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:44,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:45,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:46,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:47,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:48,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:49,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:49,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:50,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:51,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:52,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:53,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:54,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:54,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:49:55,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:56,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:57,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:58,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:58,584 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:49:59,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:49:59,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:00,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:01,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:02,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:02,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:50:02,630 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:50:02,630 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:50:02,631 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:50:03,284 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:50:03,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:04,286 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:50:04,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:04,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:05,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:06,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:07,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:08,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:09,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:09,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:10,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:11,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:12,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:13,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:14,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:14,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:15,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:16,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:17,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:18,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:19,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:19,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:20,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:21,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:22,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:23,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:24,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:24,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:25,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:26,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:27,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:28,592 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:50:28,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:29,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:30,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:30,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:31,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:32,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:33,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:34,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:50:34,584 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:50:34,584 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:50:34,585 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:50:35,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:35,311 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:50:35,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:36,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:36,312 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:50:37,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:38,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:39,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:40,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:40,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:41,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:42,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:43,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:44,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:45,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:45,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:46,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:47,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:48,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:49,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:50,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:50,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:51,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:52,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:53,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:54,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:55,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:55,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:50:56,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:57,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:58,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:50:58,589 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:50:59,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:00,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:01,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:01,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:02,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:03,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:04,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:04,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:51:04,809 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:51:04,810 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:51:04,812 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:51:05,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:05,350 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:51:06,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:06,351 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:51:06,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:07,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:08,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:09,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:10,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:11,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:11,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:12,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:13,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:14,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:15,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:16,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:16,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:17,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:18,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:19,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:20,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:21,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:21,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:22,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:23,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:24,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:25,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:26,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:26,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:27,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:28,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:28,593 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:51:29,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:30,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:31,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:32,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:32,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:33,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:34,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:35,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:36,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:36,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:51:36,523 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:51:36,524 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:51:36,529 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:51:37,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:37,376 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:51:38,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:38,377 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:51:38,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:39,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:40,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:41,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:42,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:43,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:43,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:44,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:45,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:46,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:47,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:48,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:48,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:49,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:50,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:51,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:52,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:53,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:53,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:54,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:55,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:56,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:57,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:58,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:51:58,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:51:58,597 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:51:59,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:00,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:01,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:02,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:03,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:03,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:04,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:05,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:06,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:07,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:08,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:08,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:08,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:52:08,829 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:52:08,829 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:52:08,832 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:52:09,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:09,411 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:52:10,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:10,412 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:52:11,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:12,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:13,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:13,869 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:14,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:15,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:16,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:17,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:18,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:18,869 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:19,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:20,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:21,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:22,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:23,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:23,870 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:24,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:25,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:26,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:27,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:28,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:28,601 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:52:29,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:29,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:30,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:31,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:32,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:33,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:34,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:34,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:35,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:36,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:37,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:38,248 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:39,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:39,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:40,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:41,249 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:42,201 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:52:42,202 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:52:42,203 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:52:42,208 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:52:42,450 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:52:42,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:43,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:44,452 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:52:44,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:45,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:45,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:46,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:47,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:48,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:49,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:50,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:50,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:51,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:52,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:53,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:54,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:55,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:52:55,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:56,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:57,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:58,605 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:52:58,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:52:59,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:00,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:00,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:01,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:02,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:03,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:04,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:05,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:05,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:06,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:07,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:08,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:09,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:10,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:53:10,599 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:53:10,600 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:53:10,601 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:53:10,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:11,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:11,479 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:53:11,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:12,481 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:53:12,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:13,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:14,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:15,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:15,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:16,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:17,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:18,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:19,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:20,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:20,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:22,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:23,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:24,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:25,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:25,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:26,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:27,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:28,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:28,609 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:53:29,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:30,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:31,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:31,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:32,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:33,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:34,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:35,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:36,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:36,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:37,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:38,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:38,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:53:38,766 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:53:38,766 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:53:38,768 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:53:39,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:39,516 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:53:40,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:40,517 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:53:41,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:41,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:42,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:43,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:44,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:45,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:46,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:46,807 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:47,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:48,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:49,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:50,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:51,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:51,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:52,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:53,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:54,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:55,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:56,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:56,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:53:57,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:58,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:53:58,613 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:53:59,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:00,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:01,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:02,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:02,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:03,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:04,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:05,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:06,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:07,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:07,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:08,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:08,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:54:08,780 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:54:08,780 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:54:08,781 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:54:09,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:09,546 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:54:10,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:10,547 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:54:11,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:12,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:12,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:13,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:14,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:15,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:16,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:17,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:17,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:18,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:19,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:20,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:21,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:22,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:22,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:23,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:24,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:25,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:26,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:27,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:27,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:28,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:28,618 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:54:29,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:30,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:31,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:32,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:33,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:33,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:34,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:35,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:36,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:37,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:38,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:38,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:39,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:40,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:40,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:54:40,692 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:54:40,693 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:54:40,694 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:54:41,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:41,572 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:54:42,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:42,573 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:54:43,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:43,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:44,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:45,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:46,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:47,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:48,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:48,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:49,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:50,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:51,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:52,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:53,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:53,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:54:54,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:55,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:56,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:57,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:58,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:58,621 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:54:59,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:54:59,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:00,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:01,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:02,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:03,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:04,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:04,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:05,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:06,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:07,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:08,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:09,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:55:09,175 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:55:09,175 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:55:09,176 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:55:09,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:09,599 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:55:10,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:10,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:10,600 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:55:11,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:12,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:13,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:14,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:15,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:15,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:16,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:17,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:18,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:19,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:20,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:20,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:21,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:22,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:23,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:24,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:25,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:25,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:26,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:27,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:28,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:28,624 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:55:29,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:30,579 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:30,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:31,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:32,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:33,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:34,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:35,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:35,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:36,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:37,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:38,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:55:38,566 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:55:38,567 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:55:38,568 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:55:38,634 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:55:38,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:39,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:40,635 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:55:40,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:41,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:41,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:42,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:43,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:44,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:45,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:46,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:46,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:47,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:48,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:49,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:50,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:51,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:51,708 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:52,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:53,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:54,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:55,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:56,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:55:56,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:57,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:58,628 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:55:58,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:55:59,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:00,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:01,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:01,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:02,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:03,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:04,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:05,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:06,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:06,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:07,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:08,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:09,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:10,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:11,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:11,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:12,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:13,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:13,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:56:13,800 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:56:13,801 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:56:13,802 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:56:14,668 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:56:14,669 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:56:14,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:15,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:16,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:16,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:17,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:18,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:19,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:20,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:21,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:21,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:22,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:23,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:24,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:25,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:26,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:26,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:27,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:28,632 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:56:28,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:29,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:30,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:31,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:32,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:32,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:33,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:34,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:35,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:36,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:37,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:37,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:38,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:39,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:40,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:41,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:42,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:42,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:42,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:56:42,720 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:56:42,720 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:56:42,721 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:56:43,696 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:56:43,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:44,697 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:56:44,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:45,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:46,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:47,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:47,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:48,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:49,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:50,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:51,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:52,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:52,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:53,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:54,719 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:55,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:56,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:57,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:56:57,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:58,636 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:56:58,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:56:59,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:00,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:01,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:02,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:03,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:03,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:04,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:05,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:06,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:07,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:08,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:08,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:09,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:10,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:11,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:12,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:13,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:57:13,167 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:57:13,168 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:57:13,169 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:57:13,730 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:57:13,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:14,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:14,731 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:57:14,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:15,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:16,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:17,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:18,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:19,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:19,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:20,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:21,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:22,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:23,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:24,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:24,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:25,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:26,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:27,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:28,640 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:57:28,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:29,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:29,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:30,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:31,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:32,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:33,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:34,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:34,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:35,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:36,869 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:37,869 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:38,871 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:39,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:39,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:40,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:40,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:57:40,892 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:57:40,893 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:57:40,895 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:57:41,756 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:57:41,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:42,757 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:57:42,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:43,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:44,879 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:44,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:45,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:46,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:47,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:48,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:49,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:49,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:50,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:51,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:52,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:53,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:54,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:54,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:57:55,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:56,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:57,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:58,644 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:57:58,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:57:59,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:00,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:00,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:01,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:02,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:03,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:04,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:05,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:05,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:06,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:07,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:08,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:09,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:09,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:58:09,992 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:58:09,992 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:58:09,993 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:58:10,800 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:58:10,800 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:58:10,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:11,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:11,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:12,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:13,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:14,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:15,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:16,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:16,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:17,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:18,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:19,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:20,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:21,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:21,884 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:22,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:23,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:24,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:25,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:26,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:26,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:27,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:28,648 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:58:28,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:29,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:30,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:31,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:31,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:32,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:33,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:34,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:35,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:36,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:36,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:37,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:38,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:39,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:58:39,027 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:58:39,028 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:58:39,028 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:58:39,826 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:58:39,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:40,827 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:58:40,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:41,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:42,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:42,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:43,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:44,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:45,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:46,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:47,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:47,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:48,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:49,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:50,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:51,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:52,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:52,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:53,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:54,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:55,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:56,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:57,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:58:57,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:58,651 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:58:58,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:58:59,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:00,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:01,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:02,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:02,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:03,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:04,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:05,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:06,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:07,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:07,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:08,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 13:59:08,043 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 13:59:08,044 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 13:59:08,045 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 13:59:08,857 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:59:08,857 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 13:59:08,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:09,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:10,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:11,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:12,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:13,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:13,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:14,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:15,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:16,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:17,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:18,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:18,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:19,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:20,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:21,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:22,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:23,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:23,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:24,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:25,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:26,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:27,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:28,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:28,656 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:59:28,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:29,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:30,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:33,871 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 13:59:33,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:34,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:38,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:40,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 13:59:40,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 13:59:41,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:42,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:43,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:48,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 13:59:48,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:49,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 13:59:49,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:50,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:51,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:53,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 13:59:57,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 13:59:57,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 13:59:58,660 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 13:59:58,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:59,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 13:59:59,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:04,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:06,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:00:06,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:00:06,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:07,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:08,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:09,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:14,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:15,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:00:15,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:00:16,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:17,276 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:18,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:19,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:24,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:24,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:00:26,698 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:00:28,661 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:00:29,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:29,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:29,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:30,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:31,306 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:00:31,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:32,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:33,308 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:00:33,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:34,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:35,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:35,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:36,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:37,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:38,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:39,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:40,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:40,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:41,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:42,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:43,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:44,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:45,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:45,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:46,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:47,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:48,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:49,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:50,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:50,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:51,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:52,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:53,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:54,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:55,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:55,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:00:56,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:57,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:58,644 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:00:58,665 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:00:59,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:00,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:01,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:01,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:02,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:03,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:04,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:05,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:06,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:06,668 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:07,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:08,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:09,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:10,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:11,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:11,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:12,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:13,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:14,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:15,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:16,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:16,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:17,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:18,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:19,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:20,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:21,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:21,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:22,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:26,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:28,674 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:01:31,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:36,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:41,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:43,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:01:43,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:01:43,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:44,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:45,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:46,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:46,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:47,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:48,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:49,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:50,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:51,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:51,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:52,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:53,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:54,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:55,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:56,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:56,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:01:57,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:58,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:01:58,673 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:01:59,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:00,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:01,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:02,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:02,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:03,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:04,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:05,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:06,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:07,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:07,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:08,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:09,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:10,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:11,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:12,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:12,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:13,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:14,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:15,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:16,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:17,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:17,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:18,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:19,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:20,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:21,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:22,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:22,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:23,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:24,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:25,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:26,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:27,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:27,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:28,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:28,678 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:02:29,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:30,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:31,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:32,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:32,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:33,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:34,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:35,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:36,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:37,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:37,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:38,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:39,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:40,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:41,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:42,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:42,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:43,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:44,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:45,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:46,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:47,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:47,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:48,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:49,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:50,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:51,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:52,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:52,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:53,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:54,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:55,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:56,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:57,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:57,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:02:58,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:02:58,681 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:02:59,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:00,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:01,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:02,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:02,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:03,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:04,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:05,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:06,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:07,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:07,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:08,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:09,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:10,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:11,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:12,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:12,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:13,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:14,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:15,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:16,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:17,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:17,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:18,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:19,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:20,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:21,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:22,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:22,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:23,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:24,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:25,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:03:27,687 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:28,685 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:03:32,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:37,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:42,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:47,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:52,691 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:57,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:03:58,689 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:04:03,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:06,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:04:06,156 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 14:04:06,156 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:07,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:08,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:09,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:09,192 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:04:09,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:10,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:11,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:12,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:13,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:14,195 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:04:14,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:14,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:15,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:16,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:17,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:18,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:19,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:19,459 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:20,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:21,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:22,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:23,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:24,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:24,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:25,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:26,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:27,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:28,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:28,693 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:04:29,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:29,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:30,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:31,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:32,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:33,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:34,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:34,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:35,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:36,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:37,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:38,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:38,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:04:38,707 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:04:38,707 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:04:38,708 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:04:39,220 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:04:39,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:39,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:40,221 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:04:40,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:41,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:42,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:43,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:44,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:44,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:45,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:46,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:47,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:48,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:49,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:49,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:50,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:51,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:52,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:53,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:54,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:54,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:04:55,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:56,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:57,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:58,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:04:58,697 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:04:59,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:00,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:00,699 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:01,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:02,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:03,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:04,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:05,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:05,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:06,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:07,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:08,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:09,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:10,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:10,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:11,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:11,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:05:11,987 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:05:11,987 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:05:11,988 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:05:12,255 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:05:12,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:13,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:14,257 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:05:14,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:15,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:16,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:16,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:17,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:18,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:19,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:20,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:21,027 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:21,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:22,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:23,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:24,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:25,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:26,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:26,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:27,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:28,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:28,700 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:05:29,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:30,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:31,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:31,703 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:32,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:33,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:34,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:35,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:36,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:36,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:37,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:38,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:39,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:40,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:41,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:41,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:42,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:05:42,309 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:05:42,310 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:05:42,311 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:05:42,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:43,283 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:05:43,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:44,284 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:05:44,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:45,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:46,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:47,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:47,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:48,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:49,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:50,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:51,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:52,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:52,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:53,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:54,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:55,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:56,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:57,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:05:57,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:58,703 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:05:58,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:05:59,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:00,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:01,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:02,705 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:02,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:03,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:04,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:05,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:06,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:07,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:07,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:08,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:09,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:10,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:11,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:12,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:12,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:13,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:06:13,410 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:06:13,411 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:06:13,412 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:06:13,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:14,312 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:06:14,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:15,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:16,314 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:06:16,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:17,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:18,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:18,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:19,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:20,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:21,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:22,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:23,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:23,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:24,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:25,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:26,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:27,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:28,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:28,707 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:06:28,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:29,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:30,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:31,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:32,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:33,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:33,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:34,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:35,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:36,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:37,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:38,710 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:38,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:39,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:40,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:41,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:42,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:43,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:43,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:44,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:45,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:46,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:47,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:47,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:06:47,976 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:06:47,977 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:06:47,978 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:06:48,348 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:06:48,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:49,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:49,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:50,351 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:06:50,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:51,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:52,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:53,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:54,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:54,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:55,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:56,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:57,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:58,711 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:06:58,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:06:59,713 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:06:59,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:00,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:01,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:02,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:03,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:04,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:04,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:05,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:06,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:07,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:08,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:09,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:09,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:10,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:11,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:12,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:13,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:14,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:14,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:15,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:16,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:17,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:18,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:19,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:19,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:20,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:21,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:07:21,137 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:07:21,138 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:07:21,139 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:07:21,375 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:07:21,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:22,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:23,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:24,377 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:07:24,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:25,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:25,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:26,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:27,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:28,715 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:07:28,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:29,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:30,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:30,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:31,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:32,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:33,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:34,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:35,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:35,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:36,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:37,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:38,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:39,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:40,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:40,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:41,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:42,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:43,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:44,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:45,728 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:45,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:46,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:47,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:48,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:49,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:50,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:50,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:51,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:07:51,530 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:07:51,531 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:07:51,532 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:07:52,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:52,400 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:07:53,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:54,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:54,401 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:07:55,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:56,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:56,631 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:07:57,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:58,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:07:58,719 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:07:59,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:00,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:01,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:01,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:02,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:03,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:04,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:05,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:06,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:06,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:07,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:08,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:09,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:10,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:11,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:11,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:12,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:13,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:14,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:15,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:16,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:16,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:17,051 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:18,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:19,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:20,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:21,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:08:21,037 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:08:21,037 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:08:21,038 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:08:21,428 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:08:21,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:22,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:22,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:22,429 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:08:23,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:24,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:25,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:26,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:27,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:27,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:28,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:28,723 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:08:29,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:30,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:31,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:32,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:32,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:33,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:34,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:35,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:36,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:37,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:37,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:38,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:39,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:40,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:41,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:42,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:42,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:43,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:44,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:45,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:46,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:47,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:47,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:48,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:49,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:50,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:51,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:51,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:08:51,917 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:08:51,918 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:08:51,919 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:08:52,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:52,465 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:08:52,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:53,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:54,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:54,466 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:08:55,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:56,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:57,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:57,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:08:58,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:08:58,727 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:08:59,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:00,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:01,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:02,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:03,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:03,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:04,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:05,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:06,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:07,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:08,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:08,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:09,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:10,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:11,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:12,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:13,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:13,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:14,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:15,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:16,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:17,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:18,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:18,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:19,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:20,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:21,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:22,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:22,935 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:09:22,937 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:09:22,937 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:09:22,938 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:09:23,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:23,495 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:09:23,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:24,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:24,496 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:09:25,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:26,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:27,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:28,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:28,731 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:09:29,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:29,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:30,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:31,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:32,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:33,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:34,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:34,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:35,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:36,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:37,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:38,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:39,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:39,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:40,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:41,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:42,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:43,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:44,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:44,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:45,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:46,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:47,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:48,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:49,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:49,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:50,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:51,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:52,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:52,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:09:52,900 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:09:52,901 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:09:52,902 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:09:53,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:53,521 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:09:54,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:54,522 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:09:54,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:09:55,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:56,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:57,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:58,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:09:58,736 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:09:59,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:00,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:00,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:01,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:02,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:03,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:04,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:05,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:05,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:06,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:07,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:08,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:09,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:10,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:10,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:11,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:12,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:13,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:14,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:15,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:15,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:16,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:17,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:18,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:19,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:20,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:20,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:21,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:22,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:23,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:24,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:25,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:10:25,006 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:10:25,006 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:10:25,008 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:10:25,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:25,551 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:10:26,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:26,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:26,552 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:10:27,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:28,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:28,739 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:10:29,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:30,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:31,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:31,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:32,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:33,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:34,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:35,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:36,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:36,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:37,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:38,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:39,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:40,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:41,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:41,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:42,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:43,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:44,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:45,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:46,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:46,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:47,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:48,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:49,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:50,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:51,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:51,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:52,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:53,556 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:54,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:55,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:56,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:10:56,082 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:10:56,083 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:10:56,084 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:10:56,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:56,585 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:10:57,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:10:57,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:58,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:10:58,586 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:10:58,743 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:10:59,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:00,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:01,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:02,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:02,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:03,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:04,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:05,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:06,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:07,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:07,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:08,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:09,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:10,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:11,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:12,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:12,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:13,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:14,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:15,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:16,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:17,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:17,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:18,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:19,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:20,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:21,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:22,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:22,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:23,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:24,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:25,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:25,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:11:25,761 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:11:25,761 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:11:25,762 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:11:26,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:26,616 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:11:27,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:27,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:28,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:28,617 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:11:28,747 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:11:29,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:30,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:31,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:32,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:33,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:33,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:34,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:35,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:36,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:37,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:38,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:38,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:39,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:40,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:41,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:42,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:43,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:43,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:44,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:45,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:46,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:47,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:48,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:48,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:49,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:50,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:51,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:52,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:53,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:53,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:11:54,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:55,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:56,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:11:56,010 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:11:56,010 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:11:56,011 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:11:56,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:56,642 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:11:57,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:58,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:58,644 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:11:58,751 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:11:59,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:11:59,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:00,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:01,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:02,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:03,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:04,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:04,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:05,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:06,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:07,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:08,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:09,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:09,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:10,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:11,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:12,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:13,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:14,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:14,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:15,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:16,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:17,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:18,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:19,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:19,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:20,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:21,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:22,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:23,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:24,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:24,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:25,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:12:25,193 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:12:25,193 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:12:25,194 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:12:25,674 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:12:25,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:26,674 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:12:26,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:27,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:28,755 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:12:28,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:29,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:30,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:30,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:31,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:32,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:33,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:34,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:35,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:35,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:36,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:37,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:38,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:39,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:40,758 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:40,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:41,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:42,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:43,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:44,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:45,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:45,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:46,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:47,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:48,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:49,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:50,760 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:50,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:51,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:52,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:53,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:54,775 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:55,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:12:55,753 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:12:55,754 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:12:55,755 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:12:55,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:12:56,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:56,699 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:12:56,700 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:12:56,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:57,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:58,759 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:12:58,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:12:59,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:00,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:01,761 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:01,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:02,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:03,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:04,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:05,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:06,762 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:06,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:07,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:08,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:09,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:10,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:11,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:11,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:12,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:13,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:14,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:15,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:16,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:16,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:17,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:18,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:19,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:20,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:21,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:21,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:22,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:23,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:24,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:25,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:26,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:13:26,184 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:13:26,184 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:13:26,185 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:13:26,722 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:13:26,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:27,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:27,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:28,724 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:13:28,763 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:13:28,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:29,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:30,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:31,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:32,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:32,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:33,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:34,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:35,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:36,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:37,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:37,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:38,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:39,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:40,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:41,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:42,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:42,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:43,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:44,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:45,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:46,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:47,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:47,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:48,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:49,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:50,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:51,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:52,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:52,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:53,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:54,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:55,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:56,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:57,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:13:57,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:58,768 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:13:58,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:13:59,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:00,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:01,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:14:01,264 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:14:01,265 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:14:01,266 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:14:01,752 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:14:02,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:02,752 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:14:03,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:03,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:04,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:05,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:06,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:07,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:08,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:08,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:09,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:10,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:11,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:12,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:13,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:13,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:14,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:15,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:16,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:17,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:18,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:18,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:19,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:20,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:21,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:22,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:23,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:23,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:24,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:25,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:26,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:27,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:28,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:28,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:28,773 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:14:29,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:30,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:31,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:32,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:33,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:33,783 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:34,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:35,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:14:35,088 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:14:35,089 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:14:35,090 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:14:35,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:35,783 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:14:36,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:36,784 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:14:37,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:38,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:39,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:39,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:40,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:41,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:42,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:43,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:44,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:44,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:45,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:46,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:47,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:48,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:49,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:49,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:50,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:51,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:52,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:53,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:54,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:14:54,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:55,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:56,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:57,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:58,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:58,778 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:14:59,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:14:59,781 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:00,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:01,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:02,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:02,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:15:02,688 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:15:02,689 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:15:02,690 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:15:02,808 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:15:03,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:04,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:04,810 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:15:05,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:05,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:06,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:07,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:08,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:09,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:10,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:10,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:11,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:12,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:13,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:14,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:15,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:15,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:16,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:17,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:18,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:19,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:20,731 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:20,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:21,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:22,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:23,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:24,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:25,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:25,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:26,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:27,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:28,782 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:15:28,790 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:29,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:30,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:30,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:31,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:32,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:33,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:34,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:34,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:15:34,849 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:15:34,849 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:15:34,851 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:15:35,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:35,844 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:15:35,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:36,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:36,845 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:15:37,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:38,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:39,791 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:40,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:40,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:41,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:42,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:43,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:44,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:45,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:45,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:46,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:47,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:48,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:49,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:50,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:50,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:51,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:52,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:53,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:54,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:55,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:55,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:15:56,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:57,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:58,786 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:15:58,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:15:59,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:00,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:01,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:01,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:02,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:03,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:04,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:05,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:06,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:06,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:07,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:08,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:09,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:10,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:16:10,636 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:16:10,637 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:16:10,638 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:16:10,872 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:16:11,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:12,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:12,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:12,874 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:16:13,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:14,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:15,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:16,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:17,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:17,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:18,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:19,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:20,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:21,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:22,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:22,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:23,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:24,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:25,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:26,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:27,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:27,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:28,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:28,791 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:16:29,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:30,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:31,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:32,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:32,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:33,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:34,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:35,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:36,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:37,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:37,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:38,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:39,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:39,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:16:39,990 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:16:39,990 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:16:39,991 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:16:40,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:40,898 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:16:40,899 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:16:41,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:42,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:43,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:43,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:44,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:45,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:46,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:47,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:48,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:48,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:49,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:50,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:51,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:52,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:53,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:53,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:54,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:55,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:56,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:57,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:58,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:16:58,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:16:58,793 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:16:59,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:00,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:01,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:02,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:03,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:03,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:04,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:05,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:06,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:07,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:08,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:08,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:09,138 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:10,138 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:11,138 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:12,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:17:12,007 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:17:12,008 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:17:12,009 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:17:12,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:12,924 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:17:12,925 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:17:13,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:14,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:14,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:15,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:16,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:17,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:18,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:19,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:19,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:20,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:21,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:22,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:23,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:24,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:24,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:25,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:26,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:27,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:28,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:28,797 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:17:29,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:29,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:30,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:31,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:32,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:33,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:34,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:34,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:35,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:36,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:37,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:38,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:39,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:39,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:40,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:40,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:17:40,344 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:17:40,344 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:17:40,346 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:17:40,953 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:17:40,954 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:17:41,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:42,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:43,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:44,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:45,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:45,385 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:46,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:47,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:48,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:49,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:50,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:50,385 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:51,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:52,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:53,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:54,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:55,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:55,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:17:56,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:57,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:58,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:17:58,801 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:17:59,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:00,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:00,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:01,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:02,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:03,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:04,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:05,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:05,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:06,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:07,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:08,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:09,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:10,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:10,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:11,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:11,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:18:11,270 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:18:11,270 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:18:11,271 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:18:11,985 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:18:12,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:12,986 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:18:13,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:14,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:15,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:16,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:16,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:17,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:18,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:19,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:20,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:21,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:21,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:22,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:23,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:24,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:25,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:26,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:26,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:27,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:28,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:28,805 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:18:29,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:30,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:31,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:31,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:32,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:33,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:34,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:35,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:36,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:36,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:37,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:38,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:39,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:40,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:41,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:41,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:42,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:43,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:43,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:18:43,780 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:18:43,781 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:18:43,782 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:18:44,012 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:18:44,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:45,013 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:18:45,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:46,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:46,821 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:47,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:48,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:49,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:50,227 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:51,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:51,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:52,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:53,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:54,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:55,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:56,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:56,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:18:57,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:58,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:18:58,810 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:18:59,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:00,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:01,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:02,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:02,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:03,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:04,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:05,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:06,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:07,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:07,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:08,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:09,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:10,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:11,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:12,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:12,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:13,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:14,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:14,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:19:14,778 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:19:14,778 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:19:14,779 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:19:15,039 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:19:15,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:16,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:17,041 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:19:17,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:17,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:18,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:19,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:20,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:21,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:22,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:22,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:23,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:24,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:25,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:26,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:27,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:27,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:28,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:28,813 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:19:29,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:30,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:31,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:32,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:33,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:33,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:34,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:35,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:36,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:37,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:38,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:38,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:39,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:40,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:41,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:42,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:43,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:43,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:44,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:19:44,189 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:19:44,189 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:19:44,191 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:19:44,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:45,069 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:19:45,069 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:19:45,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:46,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:47,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:48,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:49,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:49,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:50,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:51,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:52,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:53,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:54,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:19:54,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:55,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:56,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:57,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:58,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:58,817 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:19:59,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:19:59,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:00,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:01,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:02,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:03,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:04,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:04,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:05,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:06,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:07,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:08,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:09,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:09,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:10,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:11,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:12,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:13,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:14,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:14,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:15,337 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:15,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:20:15,624 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:20:15,624 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:20:15,626 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:20:16,096 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:20:16,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:17,097 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:20:17,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:18,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:19,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:20,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:20,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:21,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:22,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:23,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:24,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:25,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:25,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:26,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:27,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:28,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:28,821 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:20:29,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:30,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:30,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:31,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:32,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:33,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:34,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:35,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:35,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:36,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:37,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:38,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:39,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:40,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:40,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:41,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:42,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:43,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:44,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:45,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:20:45,110 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:20:45,110 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:20:45,112 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:20:45,126 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:20:45,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:46,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:46,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:47,127 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:20:47,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:48,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:49,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:50,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:51,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:51,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:52,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:53,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:54,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:55,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:56,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:20:56,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:57,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:58,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:20:58,825 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:20:59,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:00,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:01,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:01,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:02,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:03,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:04,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:05,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:06,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:06,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:07,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:08,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:09,659 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:10,671 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:11,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:11,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:12,660 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:13,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:14,658 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:15,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:21:15,393 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:21:15,394 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:21:15,395 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:21:15,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:16,158 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:21:16,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:17,159 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:21:17,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:17,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:18,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:19,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:20,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:21,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:22,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:22,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:23,804 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:24,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:25,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:26,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:27,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:27,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:28,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:28,829 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:21:29,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:30,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:31,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:32,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:32,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:33,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:34,805 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:35,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:36,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:37,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:37,832 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:38,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:39,806 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:40,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:41,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:42,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:42,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:43,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:44,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:45,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:45,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:21:45,957 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:21:45,958 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:21:45,959 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:21:46,191 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:21:46,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:47,192 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:21:47,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:47,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:48,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:49,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:50,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:51,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:52,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:52,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:53,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:54,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:55,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:56,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:57,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:57,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:21:58,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:21:58,834 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:21:59,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:00,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:01,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:02,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:03,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:03,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:04,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:05,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:06,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:07,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:08,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:08,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:09,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:10,815 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:11,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:12,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:13,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:13,845 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:14,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:15,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:16,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:17,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:22:17,675 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:22:17,676 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:22:17,677 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:22:18,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:18,225 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:22:19,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:19,226 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:22:19,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:20,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:21,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:22,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:23,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:24,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:24,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:25,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:26,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:27,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:28,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:28,837 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:22:29,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:29,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:30,075 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:31,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:32,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:33,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:34,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:34,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:35,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:36,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:37,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:38,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:39,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:39,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:40,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:41,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:42,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:43,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:44,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:44,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:45,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:46,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:46,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:22:46,985 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:22:46,985 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:22:46,986 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:22:47,252 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:22:47,392 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:48,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:49,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:49,254 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:22:50,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:50,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:51,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:52,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:53,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:54,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:55,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:22:55,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:56,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:57,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:58,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:22:58,841 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:22:59,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:00,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:00,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:01,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:02,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:03,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:04,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:05,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:05,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:06,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:07,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:08,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:09,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:10,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:10,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:11,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:12,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:13,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:14,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:15,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:15,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:23:15,411 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:23:15,411 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:23:15,412 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:23:16,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:16,287 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:23:16,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:17,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:17,288 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:23:18,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:19,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:20,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:21,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:21,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:22,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:23,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:24,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:25,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:26,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:26,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:27,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:28,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:28,845 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:23:29,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:30,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:31,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:31,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:32,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:33,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:34,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:35,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:36,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:36,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:37,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:38,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:39,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:40,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:41,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:41,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:42,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:43,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:44,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:45,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:23:45,024 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:23:45,025 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:23:45,026 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:23:45,311 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:23:45,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:46,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:47,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:47,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:47,312 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:23:48,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:49,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:50,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:51,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:52,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:52,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:53,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:54,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:55,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:56,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:57,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:23:57,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:58,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:23:58,849 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:23:59,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:00,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:01,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:02,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:02,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:03,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:04,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:05,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:06,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:07,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:07,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:08,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:09,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:10,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:11,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:12,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:12,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:13,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:14,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:14,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:24:14,220 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:24:14,220 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:24:14,222 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:24:14,340 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:24:15,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:15,341 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:24:16,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:17,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:18,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:18,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:19,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:20,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:21,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:22,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:23,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:23,261 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:24,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:25,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:26,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:27,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:28,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:28,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:28,853 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:24:29,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:30,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:31,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:32,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:33,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:33,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:34,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:35,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:36,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:37,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:38,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:38,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:39,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:40,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:41,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:41,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:24:41,190 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:24:41,190 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:24:41,192 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:24:41,371 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:24:42,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:43,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:43,372 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:24:44,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:44,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:45,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:46,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:47,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:48,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:49,184 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:49,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:50,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:51,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:52,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:53,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:54,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:54,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:24:55,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:56,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:57,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:58,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:58,856 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:24:59,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:24:59,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:00,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:01,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:02,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:03,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:04,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:04,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:05,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:06,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:07,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:08,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:09,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:25:09,043 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:25:09,043 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:25:09,044 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:25:09,397 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:25:09,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:10,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:10,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:11,398 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:25:11,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:12,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:13,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:14,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:15,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:15,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:16,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:17,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:18,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:19,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:20,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:20,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:21,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:22,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:23,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:24,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:25,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:25,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:26,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:27,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:28,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:28,860 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:25:29,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:30,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:30,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:31,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:32,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:33,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:34,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:35,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:35,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:36,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:37,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:38,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:39,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:40,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:40,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:41,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:42,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:43,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:25:43,027 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:25:43,027 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:25:43,029 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:25:43,425 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:25:43,425 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:25:43,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:44,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:45,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:46,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:46,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:47,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:48,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:49,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:50,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:51,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:51,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:52,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:53,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:54,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:55,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:56,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:25:56,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:57,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:58,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:25:58,864 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:25:59,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:00,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:01,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:01,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:02,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:03,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:04,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:05,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:06,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:06,871 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:07,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:08,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:09,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:10,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:11,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:11,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:12,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:26:12,123 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:26:12,124 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:26:12,126 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:26:12,454 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:26:12,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:13,455 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:26:13,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:14,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:15,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:16,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:17,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:17,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:18,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:19,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:20,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:21,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:22,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:22,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:23,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:24,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:25,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:26,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:27,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:27,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:28,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:28,868 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:26:29,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:30,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:31,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:32,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:32,871 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:33,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:34,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:35,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:36,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:37,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:37,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:38,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:39,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:40,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:41,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:26:41,322 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:26:41,322 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:26:41,326 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:26:41,482 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:26:41,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:42,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:43,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:43,483 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:26:43,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:44,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:45,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:46,732 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:47,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:48,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:48,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:49,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:50,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:51,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:52,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:53,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:53,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:54,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:55,733 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:56,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:57,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:58,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:26:58,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:26:58,871 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:26:59,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:00,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:01,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:02,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:03,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:03,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:04,734 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:05,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:06,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:07,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:08,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:08,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:09,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:10,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:11,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:11,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:27:11,944 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:27:11,944 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:27:11,946 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:27:12,507 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:27:12,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:13,508 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:27:13,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:13,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:14,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:15,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:16,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:17,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:18,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:18,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:19,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:20,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:21,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:22,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:23,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:23,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:24,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:25,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:26,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:27,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:28,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:28,876 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:27:29,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:29,878 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:30,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:31,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:32,745 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:33,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:34,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:34,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:35,754 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:36,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:37,753 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:38,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:39,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:27:39,675 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:27:39,676 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:27:39,677 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:27:40,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:40,533 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:27:40,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:40,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:41,534 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:27:41,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:42,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:43,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:44,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:45,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:45,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:46,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:47,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:48,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:49,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:50,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:50,816 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:52,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:52,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:53,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:54,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:55,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:27:55,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:56,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:57,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:58,879 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:27:58,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:27:59,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:00,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:00,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:01,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:02,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:03,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:04,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:05,885 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:05,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:06,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:07,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:08,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:09,267 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:28:09,268 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:28:09,268 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:28:09,270 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:28:09,563 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:28:09,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:10,565 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:28:10,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:11,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:11,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:12,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:13,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:14,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:15,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:16,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:16,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:17,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:18,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:19,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:20,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:21,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:21,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:22,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:23,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:24,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:25,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:26,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:26,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:27,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:28,883 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:28:28,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:29,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:30,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:31,886 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:31,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:32,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:33,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:34,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:35,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:36,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:36,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:37,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:38,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:39,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:40,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:41,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:41,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:42,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:28:42,540 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:28:42,540 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:28:42,542 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:28:42,605 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:28:42,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:43,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:44,607 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:28:44,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:45,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:46,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:47,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:47,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:48,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:49,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:50,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:51,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:52,580 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:52,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:53,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:54,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:55,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:56,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:57,581 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:28:57,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:58,887 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:28:58,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:28:59,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:00,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:01,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:02,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:02,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:03,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:04,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:05,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:06,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:07,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:07,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:08,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:09,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:10,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:11,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:12,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:12,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:13,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:29:13,017 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:29:13,018 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:29:13,019 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:29:13,632 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:29:13,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:14,633 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:29:14,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:15,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:16,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:17,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:18,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:18,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:19,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:20,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:21,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:22,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:23,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:23,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:24,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:25,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:26,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:27,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:28,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:28,891 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:29:28,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:29,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:30,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:31,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:32,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:33,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:33,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:34,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:35,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:36,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:37,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:38,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:38,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:39,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:40,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:41,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:42,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:43,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:43,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:44,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:45,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:29:45,596 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:29:45,596 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:29:45,598 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:29:45,657 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:29:45,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:46,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:47,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:48,659 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:29:48,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:49,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:49,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:50,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:51,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:52,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:53,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:54,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:54,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:55,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:56,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:57,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:58,896 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:29:58,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:29:59,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:29:59,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:00,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:01,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:02,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:03,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:04,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:04,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:06,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:06,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:08,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:09,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:09,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:10,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:11,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:12,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:13,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:14,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:14,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:15,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:16,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:16,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:30:16,737 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:30:16,738 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:30:16,739 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:30:17,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:17,682 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:30:18,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:18,683 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:30:19,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:20,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:20,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:21,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:22,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:23,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:24,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:25,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:25,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:26,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:27,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:28,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:28,899 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:30:29,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:30,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:30,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:31,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:32,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:33,170 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:34,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:35,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:35,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:36,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:37,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:38,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:39,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:40,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:40,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:41,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:42,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:43,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:44,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:45,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:45,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:46,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:47,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:48,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:30:48,027 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:30:48,028 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:30:48,029 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:30:48,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:48,717 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:30:49,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:50,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:50,720 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:30:51,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:51,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:52,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:53,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:54,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:55,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:56,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:30:56,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:57,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:58,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:30:58,903 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:30:59,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:00,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:01,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:01,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:02,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:03,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:04,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:05,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:06,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:06,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:07,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:08,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:09,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:10,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:11,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:11,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:12,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:13,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:14,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:15,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:16,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:16,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:17,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:31:17,364 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:31:17,364 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:31:17,365 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:31:17,746 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:31:17,774 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:18,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:18,747 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:31:19,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:20,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:21,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:22,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:22,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:23,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:24,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:25,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:26,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:27,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:27,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:28,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:28,907 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:31:29,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:30,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:31,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:32,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:32,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:33,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:34,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:35,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:36,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:37,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:37,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:38,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:39,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:40,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:41,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:42,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:42,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:43,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:44,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:45,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:46,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:47,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:31:47,188 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:31:47,189 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:31:47,190 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:31:47,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:47,770 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:31:48,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:48,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:48,772 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:31:49,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:50,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:51,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:52,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:53,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:53,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:54,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:55,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:56,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:57,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:58,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:31:58,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:31:58,911 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:31:59,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:00,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:01,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:02,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:03,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:03,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:04,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:05,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:06,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:07,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:08,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:08,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:09,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:10,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:11,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:12,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:13,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:13,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:14,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:15,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:16,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:17,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:32:17,506 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:32:17,507 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:32:17,508 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:32:17,803 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:32:17,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:18,803 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:32:18,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:19,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:19,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:20,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:21,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:22,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:23,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:24,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:24,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:25,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:26,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:27,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:28,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:28,915 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:32:29,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:29,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:30,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:31,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:32,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:33,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:34,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:34,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:35,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:36,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:37,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:38,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:39,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:39,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:40,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:41,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:42,915 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:43,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:44,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:44,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:45,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:46,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:47,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:48,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:32:48,126 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:32:48,127 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:32:48,127 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:32:48,832 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:32:48,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:49,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:50,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:50,833 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:32:50,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:51,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:52,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:53,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:54,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:55,167 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:32:55,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:56,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:57,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:58,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:32:58,920 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:33:00,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:00,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:01,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:02,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:03,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:04,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:05,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:05,923 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:06,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:07,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:08,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:09,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:10,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:10,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:11,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:12,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:13,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:14,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:15,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:15,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:16,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:16,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:33:16,318 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:33:16,318 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:33:16,320 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:33:16,858 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:33:17,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:18,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:18,860 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:33:19,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:20,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:21,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:21,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:22,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:23,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:24,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:25,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:26,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:26,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:27,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:28,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:28,923 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:33:29,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:30,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:31,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:31,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:32,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:33,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:34,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:35,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:36,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:36,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:37,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:38,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:39,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:40,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:41,073 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:41,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:42,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:43,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:44,074 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:45,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:33:45,047 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:33:45,048 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:33:45,050 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:33:45,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:45,899 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:33:46,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:46,900 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:33:47,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:47,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:48,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:49,188 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:50,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:51,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:52,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:52,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:53,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:54,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:55,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:56,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:57,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:33:57,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:58,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:33:58,927 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:33:59,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:00,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:01,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:02,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:02,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:03,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:04,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:05,253 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:06,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:07,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:07,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:08,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:09,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:10,254 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:11,258 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:12,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:12,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:13,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:14,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:15,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:16,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:16,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:34:16,681 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:34:16,681 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:34:16,682 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:34:16,926 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:34:17,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:18,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:18,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:18,928 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:34:19,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:20,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:21,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:22,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:23,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:23,721 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:24,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:25,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:26,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:27,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:28,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:28,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:28,931 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:34:29,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:30,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:31,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:32,263 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:33,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:33,933 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:34,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:35,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:36,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:37,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:38,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:38,934 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:39,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:40,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:41,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:42,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:43,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:43,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:44,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:45,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:46,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:47,271 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:47,909 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:34:47,910 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:34:47,911 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:34:47,912 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:34:47,955 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:34:48,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:48,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:49,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:50,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:50,957 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:34:51,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:52,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:53,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:53,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:34:54,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:55,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:56,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:57,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:58,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:58,935 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:34:59,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:34:59,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:00,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:01,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:02,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:03,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:04,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:04,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:05,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:06,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:07,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:08,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:09,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:09,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:10,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:11,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:12,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:13,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:14,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:14,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:15,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:16,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:17,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:18,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:19,330 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:19,944 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:20,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:21,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:35:21,055 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:35:21,055 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:35:21,057 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:35:21,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:21,985 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:35:22,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:22,985 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:35:23,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:24,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:25,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:25,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:26,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:27,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:28,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:28,939 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:35:29,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:30,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:30,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:31,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:32,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:33,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:34,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:35,462 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:35,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:36,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:37,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:38,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:39,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:40,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:40,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:41,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:42,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:43,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:44,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:45,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:45,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:46,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:47,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:48,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:49,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:50,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:35:50,098 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:35:50,099 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:35:50,100 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:35:50,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:51,010 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:35:51,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:51,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:52,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:53,012 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:35:53,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:54,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:55,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:56,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:35:56,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:57,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:58,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:35:58,943 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:35:59,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:00,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:01,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:01,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:02,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:03,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:04,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:05,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:06,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:06,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:07,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:08,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:09,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:10,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:11,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:11,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:12,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:13,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:14,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:15,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:16,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:16,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:17,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:18,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:19,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:20,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:21,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:21,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:36:21,902 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:36:21,903 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:36:21,904 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:36:22,044 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:36:22,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:22,942 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:23,045 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:36:23,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:24,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:25,505 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:26,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:27,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:27,943 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:28,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:28,947 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:36:29,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:30,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:31,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:32,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:32,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:33,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:34,506 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:35,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:36,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:37,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:37,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:38,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:39,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:40,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:41,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:42,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:42,950 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:43,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:44,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:45,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:46,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:47,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:47,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:48,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:49,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:50,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:51,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:36:51,069 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:36:51,069 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:36:51,071 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:36:51,072 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:36:51,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:52,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:53,075 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:36:53,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:53,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:54,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:55,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:56,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:57,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:58,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:36:58,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:36:58,951 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:36:59,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:00,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:01,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:03,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:03,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:03,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:04,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:05,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:06,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:07,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:08,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:08,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:09,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:10,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:11,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:12,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:13,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:13,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:14,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:15,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:16,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:17,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:18,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:18,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:19,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:20,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:20,788 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:37:20,790 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:37:20,790 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:37:20,791 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:37:21,099 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:37:21,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:22,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:23,101 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:37:23,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:24,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:24,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:25,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:26,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:27,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:28,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:28,953 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:37:29,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:29,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:30,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:31,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:32,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:33,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:34,622 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:34,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:35,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:36,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:37,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:38,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:39,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:39,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:40,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:41,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:42,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:43,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:44,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:44,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:45,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:46,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:47,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:48,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:49,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:49,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:50,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:51,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:52,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:53,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:53,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:37:53,980 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:37:53,981 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:37:53,982 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:37:54,127 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:37:54,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:55,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:37:55,128 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:37:55,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:56,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:57,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:58,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:37:58,958 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:37:59,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:00,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:00,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:01,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:02,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:03,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:04,747 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:05,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:05,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:06,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:07,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:08,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:09,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:10,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:10,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:11,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:12,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:13,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:14,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:15,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:15,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:16,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:17,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:18,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:19,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:20,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:20,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:21,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:38:21,150 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:38:21,150 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:38:21,151 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:38:21,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:22,152 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:38:22,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:23,153 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:38:23,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:24,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:25,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:26,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:26,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:27,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:28,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:28,962 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:38:29,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:30,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:31,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:31,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:32,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:33,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:34,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:35,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:36,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:36,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:37,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:38,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:39,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:40,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:41,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:41,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:42,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:43,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:44,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:45,667 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:46,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:46,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:47,670 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:48,674 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:49,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:50,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:51,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:51,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:52,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:38:52,560 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:38:52,560 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:38:52,561 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:38:52,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:53,178 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:38:53,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:54,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:55,179 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:38:55,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:56,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:57,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:38:57,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:58,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:38:58,966 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:38:59,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:00,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:01,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:02,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:02,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:03,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:04,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:05,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:06,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:07,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:07,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:08,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:09,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:10,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:11,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:12,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:12,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:13,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:14,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:15,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:16,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:17,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:17,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:18,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:19,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:20,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:21,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:22,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:39:22,365 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:39:22,365 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:39:22,367 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:39:22,368 ERROR gpu :3286844 [interfaces.py:monitor():142] Failed to sample metric: Not Supported 2025-01-04 14:39:22,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:23,205 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:39:23,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:23,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:24,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:25,206 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:39:25,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:26,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:27,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:28,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:28,969 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:39:28,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:29,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:30,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:31,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:32,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:33,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:33,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:34,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:35,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:36,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:37,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:38,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:38,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:39,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:40,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:41,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:42,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:43,974 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:43,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:44,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:45,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:46,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:47,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:48,979 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:48,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:49,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:50,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:51,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:52,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:53,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:39:53,224 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:39:53,225 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:39:53,226 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:39:53,231 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:39:53,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:54,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:54,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:55,233 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:39:55,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:56,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:57,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:58,974 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:39:58,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:39:59,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:39:59,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:00,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:01,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:02,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:03,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:04,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:04,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:05,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:06,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:07,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:08,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:09,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:09,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:10,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:11,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:12,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:13,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:14,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:14,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:15,994 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:16,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:17,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:18,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:19,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:19,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:20,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:40:20,088 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:40:20,088 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:40:20,090 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:40:20,253 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:40:21,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:21,254 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:40:21,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:22,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:23,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:24,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:25,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:25,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:26,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:27,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:28,978 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:40:28,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:30,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:30,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:31,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:32,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:33,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:34,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:35,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:35,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:36,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:37,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:38,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:39,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:40,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:40,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:41,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:42,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:43,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:44,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:45,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:45,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:46,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:47,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:48,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:49,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:49,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:40:49,716 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:40:49,717 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:40:49,718 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:40:50,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:50,279 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:40:51,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:51,280 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:40:51,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:52,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:53,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:54,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:55,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:56,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:56,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:40:57,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:58,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:40:58,982 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:40:59,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:00,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:01,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:01,984 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:02,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:03,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:04,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:05,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:06,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:06,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:07,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:08,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:09,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:10,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:11,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:11,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:12,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:13,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:14,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:15,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:16,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:16,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:17,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:18,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:41:18,003 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:41:18,004 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:41:18,005 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:41:18,307 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:41:18,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:19,308 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:41:19,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:20,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:21,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:22,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:22,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:23,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:24,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:25,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:26,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:27,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:27,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:28,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:28,985 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:41:29,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:30,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:31,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:32,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:32,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:33,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:34,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:35,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:36,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:37,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:37,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:38,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:39,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:40,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:41,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:42,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:43,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:43,453 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:44,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:45,458 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:46,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:47,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:47,998 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:48,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:48,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:41:48,686 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:41:48,686 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:41:48,687 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:41:49,339 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:41:49,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:50,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:51,341 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:41:51,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:52,455 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:53,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:53,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:54,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:55,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:56,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:57,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:58,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:41:58,727 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:41:58,989 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:41:59,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:00,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:01,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:02,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:03,456 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:03,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:04,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:05,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:06,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:07,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:08,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:08,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:09,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:10,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:11,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:12,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:13,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:13,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:14,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:15,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:16,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:17,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:18,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:18,999 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:19,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:20,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:42:20,014 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:42:20,014 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:42:20,015 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:42:20,365 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:42:20,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:21,366 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:42:21,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:22,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:23,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:24,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:24,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:25,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:26,467 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:27,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:28,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:28,994 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:42:29,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:29,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:30,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:31,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:32,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:33,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:34,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:34,996 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:35,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:36,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:37,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:38,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:39,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:39,997 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:40,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:41,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:42,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:43,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:44,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:45,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:45,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:46,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:47,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:48,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:48,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:42:48,841 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:42:48,842 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:42:48,843 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:42:49,401 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:42:49,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:50,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:50,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:51,403 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:42:51,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:52,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:53,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:54,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:55,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:55,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:42:56,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:57,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:58,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:42:58,998 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:42:59,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:00,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:01,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:01,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:02,475 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:03,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:04,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:05,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:06,000 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:06,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:07,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:08,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:09,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:10,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:11,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:11,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:12,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:13,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:14,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:15,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:16,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:16,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:17,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:18,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:19,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:20,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:21,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:21,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:22,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:22,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:43:22,850 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:43:22,851 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:43:22,852 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:43:23,431 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:43:23,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:24,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:25,433 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:43:25,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:26,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:26,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:27,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:28,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:29,002 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:43:29,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:30,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:31,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:32,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:32,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:33,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:34,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:35,653 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:36,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:37,005 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:37,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:38,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:39,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:40,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:41,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:42,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:42,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:43,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:44,557 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:45,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:46,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:47,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:47,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:48,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:49,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:50,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:51,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:52,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:52,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:52,987 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:43:52,988 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:43:52,988 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:43:52,989 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:43:53,460 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:43:53,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:54,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:55,461 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:43:55,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:56,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:57,028 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:43:57,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:58,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:43:59,005 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:43:59,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:00,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:01,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:02,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:03,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:03,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:04,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:05,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:06,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:07,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:08,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:08,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:09,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:10,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:11,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:12,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:13,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:13,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:14,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:15,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:16,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:17,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:18,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:18,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:19,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:20,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:21,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:22,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:23,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:23,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:24,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:25,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:25,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:44:25,819 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:44:25,819 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:44:25,820 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:44:26,491 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:44:26,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:27,492 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:44:27,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:28,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:28,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:29,009 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:44:29,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:30,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:31,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:32,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:33,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:34,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:34,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:35,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:36,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:37,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:38,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:39,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:39,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:40,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:41,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:42,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:43,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:44,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:44,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:45,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:46,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:47,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:48,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:49,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:49,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:50,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:51,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:52,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:53,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:54,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:44:54,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:55,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:56,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:57,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:58,255 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:44:58,256 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:44:58,256 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:44:58,257 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:44:58,518 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:44:58,666 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:44:59,014 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:44:59,519 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:44:59,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:00,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:00,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:01,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:02,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:03,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:04,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:05,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:05,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:06,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:07,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:08,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:09,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:10,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:10,663 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:11,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:12,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:13,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:14,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:15,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:15,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:16,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:17,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:18,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:19,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:20,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:20,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:21,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:22,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:23,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:24,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:25,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:25,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:26,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:27,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:28,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:45:28,497 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:45:28,498 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:45:28,499 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:45:28,548 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:45:28,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:29,018 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:45:29,549 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:45:29,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:30,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:31,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:31,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:32,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:33,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:34,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:35,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:36,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:36,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:37,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:38,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:39,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:40,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:41,021 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:41,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:42,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:43,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:44,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:45,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:46,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:46,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:47,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:48,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:49,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:50,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:51,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:51,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:52,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:53,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:54,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:55,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:56,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:45:56,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:57,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:58,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:45:59,026 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:45:59,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:00,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:46:00,312 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:46:00,312 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:46:00,313 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:46:00,579 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:46:00,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:01,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:01,580 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:46:01,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:02,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:03,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:04,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:05,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:06,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:06,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:07,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:08,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:09,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:10,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:11,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:11,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:12,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:13,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:14,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:15,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:16,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:16,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:17,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:18,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:19,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:20,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:21,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:21,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:22,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:23,921 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:24,922 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:25,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:26,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:26,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:27,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:28,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:29,029 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:46:29,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:30,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:31,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:46:31,019 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:46:31,019 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:46:31,020 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:46:31,610 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:46:31,611 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:46:31,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:32,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:32,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:33,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:34,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:35,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:36,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:37,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:37,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:38,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:39,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:40,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:41,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:42,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:42,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:43,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:44,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:46,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:47,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:47,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:48,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:49,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:50,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:51,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:52,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:52,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:53,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:54,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:55,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:56,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:57,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:57,063 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:46:58,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:46:59,029 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:46:59,038 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:00,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:01,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:02,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:03,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:03,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:04,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:05,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:05,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:47:05,551 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:47:05,552 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:47:05,557 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:47:05,638 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:47:06,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:07,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:07,640 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:47:08,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:08,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:09,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:10,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:11,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:12,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:13,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:13,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:14,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:15,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:16,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:17,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:18,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:18,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:19,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:20,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:21,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:22,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:23,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:23,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:24,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:25,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:26,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:27,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:28,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:28,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:29,039 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:47:29,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:30,052 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:31,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:32,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:33,054 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:34,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:34,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:34,706 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:47:34,707 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:47:34,707 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:47:34,708 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:47:35,117 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:35,677 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:47:35,677 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:47:36,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:37,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:38,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:39,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:39,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:40,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:41,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:42,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:43,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:44,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:44,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:45,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:46,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:47,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:48,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:49,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:49,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:50,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:51,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:52,131 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:53,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:54,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:54,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:47:55,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:56,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:57,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:58,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:47:59,056 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:47:59,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:00,059 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:00,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:01,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:02,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:03,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:03,695 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:48:03,696 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:48:03,696 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:48:03,697 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:48:03,710 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:48:04,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:05,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:05,712 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:48:05,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:06,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:07,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:08,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:09,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:10,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:10,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:11,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:12,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:13,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:14,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:15,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:15,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:16,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:17,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:18,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:19,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:20,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:20,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:21,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:22,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:23,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:24,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:25,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:25,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:26,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:27,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:28,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:29,045 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:48:29,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:30,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:31,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:31,150 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:32,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:32,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:48:32,950 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:48:32,950 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:48:32,953 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:48:33,738 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:48:33,739 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:48:34,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:35,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:36,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:36,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:37,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:38,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:39,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:40,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:41,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:41,991 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:42,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:43,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:44,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:45,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:46,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:46,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:47,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:48,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:49,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:50,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:51,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:51,992 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:52,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:53,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:54,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:55,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:56,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:56,993 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:48:57,034 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:58,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:59,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:48:59,050 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:49:00,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:01,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:02,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:02,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:03,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:04,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:04,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:49:04,204 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:49:04,205 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:49:04,210 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:49:04,763 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:49:05,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:05,764 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:49:06,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:07,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:07,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:08,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:09,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:10,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:11,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:12,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:12,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:13,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:14,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:15,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:16,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:17,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:17,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:18,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:19,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:20,044 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:21,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:22,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:22,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:23,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:24,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:25,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:26,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:27,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:27,247 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:28,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:29,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:29,061 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:49:30,050 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:31,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:32,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:33,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:33,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:34,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:34,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:49:34,940 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:49:34,940 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:49:34,945 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:49:35,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:35,788 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:49:35,789 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:49:36,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:37,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:38,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:38,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:39,081 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:40,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:41,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:42,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:43,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:43,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:44,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:45,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:46,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:47,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:48,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:48,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:49,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:50,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:51,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:52,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:53,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:53,982 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:54,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:55,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:56,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:57,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:58,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:49:58,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:49:59,054 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:49:59,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:00,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:01,094 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:02,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:03,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:04,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:04,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:05,088 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:06,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:06,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:50:06,359 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:50:06,359 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:50:06,362 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:50:06,816 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:50:07,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:07,817 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:50:08,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:09,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:09,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:10,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:11,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:12,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:13,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:14,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:14,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:15,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:16,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:17,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:18,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:19,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:19,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:20,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:21,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:22,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:23,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:24,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:24,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:25,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:26,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:27,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:28,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:29,058 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:50:29,091 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:30,064 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:30,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:31,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:32,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:33,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:34,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:35,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:35,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:35,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:50:35,333 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:50:35,333 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:50:35,334 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:50:35,850 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:50:35,850 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:50:36,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:37,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:38,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:39,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:40,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:40,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:41,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:42,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:43,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:44,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:45,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:45,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:46,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:47,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:48,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:49,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:50,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:50,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:51,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:52,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:53,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:54,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:55,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:55,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:50:56,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:57,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:58,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:50:59,070 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:50:59,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:00,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:01,071 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:01,109 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:02,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:03,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:04,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:04,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:51:04,787 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:51:04,788 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:51:04,789 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:51:04,878 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:51:05,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:05,879 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:51:06,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:06,826 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:07,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:08,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:09,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:10,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:11,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:11,827 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:12,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:13,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:14,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:15,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:16,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:16,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:17,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:18,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:19,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:20,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:21,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:21,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:22,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:23,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:24,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:25,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:26,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:26,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:27,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:28,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:29,079 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:51:29,329 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:30,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:31,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:32,087 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:32,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:33,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:34,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:34,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:51:34,814 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:51:34,815 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:51:34,816 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:51:34,909 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:51:35,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:36,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:36,911 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:51:37,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:37,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:38,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:39,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:40,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:41,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:42,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:42,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:43,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:44,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:45,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:46,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:47,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:47,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:48,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:49,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:50,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:51,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:52,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:52,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:53,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:54,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:55,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:56,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:57,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:57,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:51:58,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:51:59,072 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:51:59,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:00,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:01,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:02,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:03,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:03,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:04,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:05,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:05,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:52:05,596 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:52:05,597 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:52:05,602 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:52:05,944 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:52:06,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:06,945 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:52:07,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:08,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:08,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:09,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:10,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:11,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:12,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:13,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:13,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:14,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:15,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:16,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:17,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:18,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:18,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:19,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:20,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:21,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:22,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:23,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:23,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:24,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:25,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:26,374 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:27,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:28,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:28,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:29,080 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:52:29,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:30,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:31,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:32,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:33,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:34,097 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:34,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:35,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:52:35,081 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:52:35,081 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:52:35,083 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:52:35,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:35,973 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:52:36,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:36,974 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:52:37,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:38,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:39,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:39,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:40,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:41,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:42,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:43,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:44,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:44,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:45,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:46,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:47,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:48,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:49,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:49,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:50,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:51,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:52,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:53,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:54,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:52:54,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:55,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:56,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:57,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:58,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:52:59,080 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:52:59,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:00,082 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:00,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:01,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:02,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:03,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:04,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:05,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:05,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:06,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:06,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:53:06,788 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:53:06,789 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:53:06,794 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:53:07,009 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:53:07,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:08,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:09,010 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:53:09,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:10,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:10,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:11,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:12,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:13,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:14,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:15,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:15,829 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:16,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:17,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:18,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:19,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:20,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:20,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:21,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:22,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:23,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:24,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:25,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:25,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:26,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:27,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:28,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:29,084 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:53:29,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:30,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:31,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:31,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:32,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:33,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:34,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:35,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:36,096 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:36,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:36,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:53:36,857 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:53:36,858 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:53:36,861 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:53:37,039 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:53:37,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:38,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:39,040 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:53:39,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:40,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:41,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:41,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:42,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:43,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:44,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:45,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:46,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:46,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:47,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:48,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:49,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:50,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:51,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:51,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:52,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:53,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:54,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:55,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:56,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:56,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:53:57,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:58,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:53:59,088 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:53:59,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:00,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:01,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:02,090 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:02,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:03,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:04,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:05,722 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:06,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:07,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:07,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:08,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:09,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:10,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:54:10,201 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:54:10,202 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:54:10,207 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:54:10,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:11,068 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:54:11,723 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:12,240 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:12,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:13,069 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:54:13,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:14,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:15,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:16,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:17,241 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:17,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:18,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:19,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:20,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:21,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:22,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:22,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:23,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:24,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:25,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:26,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:27,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:27,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:28,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:29,092 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:54:29,725 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:30,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:31,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:32,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:33,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:33,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:34,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:35,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:36,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:37,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:38,095 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:38,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:39,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:40,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:41,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:42,735 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:43,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:43,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:44,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:45,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:46,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:54:46,246 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:54:46,247 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:54:46,252 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:54:46,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:47,093 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:54:47,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:48,287 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:48,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:49,094 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:54:49,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:50,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:51,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:52,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:53,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:53,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:54,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:55,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:56,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:57,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:58,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:54:58,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:54:59,096 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:54:59,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:00,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:01,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:02,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:03,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:04,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:04,737 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:05,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:06,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:07,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:08,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:09,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:09,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:10,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:11,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:12,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:13,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:14,100 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:14,738 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:15,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:16,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:17,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:18,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:19,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:19,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:20,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:21,739 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:55:21,741 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:55:21,742 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:55:21,747 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:55:21,780 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:22,123 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:55:22,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:23,125 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:55:23,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:24,740 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:24,781 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:25,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:26,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:27,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:28,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:29,100 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:55:29,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:30,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:30,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:31,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:32,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:33,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:34,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:35,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:35,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:36,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:37,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:38,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:39,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:40,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:40,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:41,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:42,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:43,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:44,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:45,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:45,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:46,837 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:47,839 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:48,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:49,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:50,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:50,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:51,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:55:51,176 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:55:51,177 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:55:51,178 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:55:51,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:52,147 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:55:52,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:53,148 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:55:53,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:54,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:55,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:55:55,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:56,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:57,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:58,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:55:59,104 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:55:59,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:00,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:01,106 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:01,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:02,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:03,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:04,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:05,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:06,107 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:06,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:07,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:08,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:09,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:10,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:11,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:11,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:12,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:13,842 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:14,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:15,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:16,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:16,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:17,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:18,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:19,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:20,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:56:20,405 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:56:20,405 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:56:20,407 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:56:20,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:21,173 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:56:21,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:21,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:22,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:23,175 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:56:23,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:24,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:25,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:26,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:26,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:27,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:28,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:29,108 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:56:29,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:30,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:31,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:32,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:32,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:33,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:34,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:35,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:36,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:37,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:37,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:38,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:39,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:40,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:41,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:42,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:42,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:43,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:44,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:45,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:46,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:47,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:47,868 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:48,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:49,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:50,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:51,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:52,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:52,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:53,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:56:53,436 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:56:53,437 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:56:53,439 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:56:53,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:54,200 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:56:54,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:55,201 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:56:55,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:56,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:57,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:56:57,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:58,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:56:59,112 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:56:59,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:00,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:01,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:02,857 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:03,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:03,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:04,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:05,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:06,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:07,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:08,115 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:08,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:09,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:10,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:11,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:12,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:13,116 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:13,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:14,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:15,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:16,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:17,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:18,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:18,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:19,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:20,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:21,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:57:21,356 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:57:21,357 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:57:21,358 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:57:21,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:22,224 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:57:22,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:23,225 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:57:23,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:23,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:24,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:25,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:26,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:27,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:28,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:28,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:29,116 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:57:29,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:30,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:31,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:32,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:33,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:34,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:34,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:35,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:36,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:37,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:38,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:39,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:39,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:40,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:41,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:42,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:43,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:44,120 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:44,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:45,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:46,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:47,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:48,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:49,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:49,899 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:50,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:51,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:52,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:53,907 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:54,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:57:54,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:57:54,743 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:57:54,743 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:57:54,745 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:57:55,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:55,257 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:57:56,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:57,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:57,259 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:57:58,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:57:59,119 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:57:59,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:00,121 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:00,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:01,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:02,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:03,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:04,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:05,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:05,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:06,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:07,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:08,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:09,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:10,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:10,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:11,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:12,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:13,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:14,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:15,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:15,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:16,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:17,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:18,167 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:19,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:20,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:20,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:21,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:22,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:22,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:58:22,199 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:58:22,200 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:58:22,201 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:58:22,286 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:58:23,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:23,287 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:58:24,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:25,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:25,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:26,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:27,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:28,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:29,122 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:58:29,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:30,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:31,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:31,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:32,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:33,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:34,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:35,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:36,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:36,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:37,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:38,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:39,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:40,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:41,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:41,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:42,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:43,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:44,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:45,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:46,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:46,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:47,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:48,176 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:49,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:50,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:51,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:51,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:52,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:53,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:53,688 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:58:53,690 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:58:53,691 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:58:53,692 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:58:54,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:54,317 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:58:55,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:55,318 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:58:56,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:56,730 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:58:57,172 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:58,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:58:59,126 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:58:59,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:00,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:01,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:02,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:02,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:03,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:04,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:05,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:06,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:07,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:07,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:08,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:09,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:10,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:11,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:12,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:12,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:13,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:14,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:15,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:16,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:17,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:17,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:18,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:19,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:20,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:21,183 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:21,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:59:21,716 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:59:21,716 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:59:21,718 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:59:22,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:22,343 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:59:22,756 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:23,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:23,344 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:59:24,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:25,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:26,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:27,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:27,757 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:28,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:29,130 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:59:29,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:30,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:31,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:32,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:33,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:33,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:34,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:35,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:36,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:37,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:38,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:38,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:39,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:40,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:41,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:42,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:43,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:43,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:44,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:45,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:46,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:47,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:48,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:48,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:49,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:50,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:51,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:51,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 14:59:51,636 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 14:59:51,637 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 14:59:51,638 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 14:59:52,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:52,367 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 14:59:53,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:53,367 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 14:59:53,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:54,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:55,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:56,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:57,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:58,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 14:59:58,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 14:59:59,134 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 14:59:59,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:00,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:01,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:02,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:03,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:04,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:04,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:05,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:06,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:07,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:08,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:09,137 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:09,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:10,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:11,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:12,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:13,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:14,138 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:14,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:15,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:16,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:17,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:18,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:19,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:19,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:20,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:21,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:22,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:23,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:24,068 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:00:24,069 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:00:24,070 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:00:24,072 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:00:24,401 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:00:24,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:25,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:25,402 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:00:25,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:26,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:27,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:28,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:29,138 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:00:29,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:30,140 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:30,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:31,464 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:32,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:33,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:34,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:35,141 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:35,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:36,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:37,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:38,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:39,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:40,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:40,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:41,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:42,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:43,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:44,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:45,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:45,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:46,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:47,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:48,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:49,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:50,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:50,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:51,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:52,476 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:53,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:54,086 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:00:54,087 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:00:54,088 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:00:54,089 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:00:54,427 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:00:54,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:55,429 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:00:55,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:56,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:00:56,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:57,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:58,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:00:59,141 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:00:59,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:00,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:01,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:01,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:02,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:03,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:04,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:05,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:06,144 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:06,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:07,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:08,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:09,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:10,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:11,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:11,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:12,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:13,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:14,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:15,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:16,145 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:16,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:17,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:18,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:19,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:20,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:21,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:21,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:22,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:23,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:23,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:01:23,527 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:01:23,527 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:01:23,528 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:01:24,458 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:01:24,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:25,459 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:01:25,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:26,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:26,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:27,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:28,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:29,145 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:01:29,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:30,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:31,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:32,147 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:32,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:33,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:34,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:35,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:36,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:37,148 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:37,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:38,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:39,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:40,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:41,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:42,149 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:42,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:43,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:44,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:45,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:46,501 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:47,156 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:47,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:48,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:49,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:50,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:51,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:52,154 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:52,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:52,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:01:52,666 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:01:52,667 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:01:52,668 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:01:53,480 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:01:53,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:54,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:55,481 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:01:55,507 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:56,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:57,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:57,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:01:58,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:01:59,149 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:01:59,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:00,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:01,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:02,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:03,151 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:03,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:04,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:05,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:06,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:07,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:08,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:08,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:09,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:10,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:11,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:12,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:13,152 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:13,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:14,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:15,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:16,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:17,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:18,153 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:18,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:19,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:20,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:21,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:22,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:23,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:23,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:24,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:24,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:02:24,989 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:02:24,990 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:02:24,991 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:02:25,508 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:02:25,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:26,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:27,510 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:02:27,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:28,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:29,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:29,154 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:02:29,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:30,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:31,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:32,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:33,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:34,156 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:34,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:35,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:36,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:37,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:38,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:39,157 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:39,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:40,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:41,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:42,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:43,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:44,158 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:44,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:45,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:46,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:47,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:48,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:49,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:49,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:50,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:51,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:52,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:53,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:54,168 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:02:54,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:54,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:02:54,890 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:02:54,890 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:02:54,891 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:02:55,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:55,538 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:02:56,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:57,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:57,539 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:02:58,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:02:59,157 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:02:59,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:00,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:00,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:01,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:02,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:03,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:04,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:05,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:05,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:06,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:07,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:08,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:09,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:10,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:10,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:11,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:12,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:13,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:14,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:15,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:15,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:16,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:17,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:18,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:19,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:20,162 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:20,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:21,523 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:22,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:23,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:24,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:25,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:25,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:26,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:27,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:03:27,445 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:03:27,445 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:03:27,447 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:03:27,567 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:03:27,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:28,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:29,161 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:03:29,569 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:03:29,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:30,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:31,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:31,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:32,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:33,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:34,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:35,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:36,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:36,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:37,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:38,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:39,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:40,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:41,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:41,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:42,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:43,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:44,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:45,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:46,165 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:46,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:47,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:48,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:49,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:50,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:51,166 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:51,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:52,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:53,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:54,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:55,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:56,171 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:03:56,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:57,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:03:57,540 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:03:57,541 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:03:57,542 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:03:57,597 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:03:57,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:58,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:03:59,171 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:03:59,599 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:03:59,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:00,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:01,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:01,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:02,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:03,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:04,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:05,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:06,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:06,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:07,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:08,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:09,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:10,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:11,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:11,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:12,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:13,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:14,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:15,778 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:16,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:16,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:17,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:18,779 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:19,785 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:20,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:21,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:21,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:22,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:23,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:24,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:25,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:04:25,405 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:04:25,405 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:04:25,407 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:04:25,621 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:04:25,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:26,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:26,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:27,623 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:04:27,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:28,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:29,171 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:04:29,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:30,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:31,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:32,173 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:32,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:33,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:34,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:35,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:36,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:37,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:37,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:38,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:39,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:40,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:41,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:42,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:42,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:43,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:44,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:45,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:46,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:47,175 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:47,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:48,846 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:49,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:50,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:51,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:52,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:52,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:53,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:54,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:04:54,086 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:04:54,087 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:04:54,088 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:04:54,651 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:04:54,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:55,652 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:04:55,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:56,840 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:57,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:58,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:04:58,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:04:59,175 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:04:59,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:00,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:01,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:02,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:03,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:03,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:04,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:05,841 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:06,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:07,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:08,178 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:08,910 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:09,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:10,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:11,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:12,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:13,179 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:13,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:14,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:15,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:16,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:17,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:18,180 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:18,913 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:19,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:20,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:21,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:22,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:23,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:23,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:24,308 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:05:24,309 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:05:24,309 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:05:24,310 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:05:24,676 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:05:24,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:25,678 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:05:25,918 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:26,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:27,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:28,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:28,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:29,179 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:05:29,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:30,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:31,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:32,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:33,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:34,181 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:34,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:35,919 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:36,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:37,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:38,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:39,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:39,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:40,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:41,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:42,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:43,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:44,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:44,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:45,920 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:46,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:47,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:48,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:49,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:49,928 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:50,927 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:51,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:52,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:52,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:05:52,941 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:05:52,941 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:05:52,942 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:05:53,708 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:05:53,708 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:05:53,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:54,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:54,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:05:55,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:56,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:57,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:58,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:05:59,183 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:05:59,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:00,185 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:00,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:01,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:02,936 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:03,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:05,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:05,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:06,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:07,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:08,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:09,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:10,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:10,186 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:11,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:12,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:13,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:14,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:15,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:15,187 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:16,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:17,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:18,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:19,007 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:20,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:20,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:21,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:22,010 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:22,707 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:06:22,709 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:06:22,709 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:06:22,711 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:06:22,736 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:06:23,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:23,737 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:06:24,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:25,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:25,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:26,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:27,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:28,124 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:29,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:29,187 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:06:30,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:31,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:31,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:32,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:33,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:34,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:35,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:36,125 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:36,189 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:37,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:38,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:39,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:40,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:41,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:41,190 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:42,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:43,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:44,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:45,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:46,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:46,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:47,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:48,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:49,127 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:50,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:51,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:51,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:52,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:53,132 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:54,069 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:06:54,070 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:06:54,071 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:06:54,072 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:06:54,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:54,770 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:06:55,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:55,771 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:06:56,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:57,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:06:57,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:58,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:06:59,191 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:06:59,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:00,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:01,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:02,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:02,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:03,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:04,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:05,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:06,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:07,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:07,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:08,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:09,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:10,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:11,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:12,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:12,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:13,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:14,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:15,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:16,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:17,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:17,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:18,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:19,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:20,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:21,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:22,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:22,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:23,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:07:23,220 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:07:23,220 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:07:23,222 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:07:23,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:23,794 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:07:24,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:25,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:25,796 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:07:26,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:27,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:27,260 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:28,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:29,195 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:07:29,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:30,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:31,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:32,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:33,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:33,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:34,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:35,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:36,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:37,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:38,198 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:38,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:39,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:40,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:41,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:42,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:43,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:43,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:44,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:45,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:46,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:47,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:48,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:48,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:49,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:50,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:51,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:52,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:53,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:53,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:54,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:07:54,086 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:07:54,086 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:07:54,088 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:07:54,494 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:54,829 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:07:55,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:55,830 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:07:56,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:57,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:58,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:07:59,126 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:07:59,199 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:07:59,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:00,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:01,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:02,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:03,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:04,201 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:04,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:05,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:06,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:07,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:08,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:09,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:09,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:10,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:11,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:12,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:13,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:14,203 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:14,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:15,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:16,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:17,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:18,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:19,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:19,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:20,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:21,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:22,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:23,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:24,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:24,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:25,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:25,782 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:08:25,784 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:08:25,784 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:08:25,785 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:08:25,859 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:08:26,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:27,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:27,861 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:08:28,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:29,203 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:08:29,236 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:30,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:30,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:31,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:32,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:33,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:34,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:35,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:35,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:36,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:37,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:38,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:39,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:40,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:40,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:41,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:42,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:43,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:44,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:45,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:45,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:46,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:47,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:48,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:49,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:50,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:50,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:51,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:52,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:53,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:54,246 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:54,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:08:54,942 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:08:54,942 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:08:54,944 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:08:55,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:55,883 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:08:55,884 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:08:55,981 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:08:56,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:57,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:58,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:08:59,205 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:08:59,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:00,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:01,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:01,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:02,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:03,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:04,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:05,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:06,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:06,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:07,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:08,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:09,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:10,441 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:11,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:11,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:12,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:13,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:14,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:15,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:16,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:16,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:17,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:18,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:19,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:20,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:21,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:21,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:22,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:23,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:24,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:25,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:26,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:09:26,217 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:09:26,217 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:09:26,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:26,218 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:09:26,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:26,910 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:09:27,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:27,911 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:09:28,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:29,209 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:09:29,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:30,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:31,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:32,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:32,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:33,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:34,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:35,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:36,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:37,212 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:37,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:38,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:39,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:40,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:41,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:42,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:42,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:43,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:44,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:45,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:46,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:47,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:47,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:48,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:49,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:50,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:51,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:52,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:52,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:53,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:54,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:55,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:56,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:56,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:09:56,949 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:09:56,950 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:09:56,951 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:09:57,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:57,939 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:09:57,939 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:09:57,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:09:58,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:09:59,213 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:09:59,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:00,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:01,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:02,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:03,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:03,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:04,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:05,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:06,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:07,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:08,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:08,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:09,637 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:10,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:11,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:12,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:13,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:13,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:14,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:15,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:16,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:17,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:18,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:18,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:19,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:20,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:21,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:22,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:23,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:23,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:24,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:25,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:26,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:27,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:10:27,400 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:10:27,400 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:10:27,402 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:10:27,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:27,965 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:10:28,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:28,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:29,218 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:10:29,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:29,967 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:10:30,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:31,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:32,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:33,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:34,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:34,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:35,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:36,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:37,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:38,801 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:39,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:39,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:40,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:41,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:42,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:43,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:44,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:44,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:45,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:46,802 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:47,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:48,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:49,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:49,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:50,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:51,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:52,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:53,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:54,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:10:54,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:55,808 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:56,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:10:56,352 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:10:56,353 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:10:56,354 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:10:56,809 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:56,994 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:10:57,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:57,995 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:10:58,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:10:59,221 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:10:59,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:00,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:00,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:01,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:02,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:03,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:04,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:05,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:05,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:06,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:07,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:08,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:09,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:10,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:10,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:11,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:12,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:13,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:14,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:15,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:15,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:16,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:17,813 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:18,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:19,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:20,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:20,819 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:21,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:22,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:23,822 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:24,817 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:25,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:25,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:11:25,550 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:11:25,550 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:11:25,552 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:11:25,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:26,028 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:11:26,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:27,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:28,030 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:11:28,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:29,226 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:11:29,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:30,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:31,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:31,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:32,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:33,959 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:34,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:35,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:36,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:36,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:37,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:38,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:39,960 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:40,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:41,229 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:41,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:42,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:43,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:44,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:45,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:46,230 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:46,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:47,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:48,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:49,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:50,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:51,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:51,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:52,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:53,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:54,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:55,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:56,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:11:56,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:57,828 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:11:57,829 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:11:57,829 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:11:57,830 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:11:58,057 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:11:58,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:58,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:11:59,230 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:11:59,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:00,059 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:12:00,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:01,975 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:02,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:02,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:03,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:04,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:05,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:06,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:07,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:07,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:08,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:09,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:10,976 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:11,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:12,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:12,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:13,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:14,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:15,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:16,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:17,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:17,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:18,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:19,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:20,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:21,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:22,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:22,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:23,978 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:24,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:25,986 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:26,985 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:27,242 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:27,990 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:28,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:29,231 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:12:29,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:30,164 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:12:30,165 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:12:30,166 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:12:30,167 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:12:30,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:31,084 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:12:31,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:32,085 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:12:32,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:33,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:33,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:34,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:35,988 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:36,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:37,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:38,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:38,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:39,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:40,989 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:42,182 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:43,092 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:43,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:44,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:45,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:46,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:47,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:48,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:48,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:49,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:50,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:51,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:52,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:53,093 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:53,208 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:54,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:55,101 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:56,098 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:57,099 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:58,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:58,213 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:12:59,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:12:59,233 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:12:59,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:12:59,431 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:12:59,431 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:12:59,433 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:13:00,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:00,109 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:13:00,109 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:13:01,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:02,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:03,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:03,470 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:04,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:05,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:06,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:07,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:08,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:08,471 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:09,102 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:10,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:11,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:12,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:13,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:13,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:14,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:15,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:16,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:17,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:18,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:18,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:19,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:20,103 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:21,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:22,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:23,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:23,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:24,104 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:25,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:26,110 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:27,111 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:28,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:28,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:29,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:29,234 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:13:30,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:30,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:13:30,283 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:13:30,283 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:13:30,285 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:13:31,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:31,138 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:13:32,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:32,139 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:13:33,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:34,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:34,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:35,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:36,112 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:37,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:38,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:39,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:39,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:40,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:41,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:42,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:43,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:44,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:44,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:45,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:46,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:47,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:48,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:49,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:49,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:50,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:51,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:52,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:53,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:54,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:54,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:13:55,114 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:56,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:57,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:58,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:59,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:13:59,242 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:14:00,123 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:00,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:01,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:01,174 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:14:01,175 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:14:01,175 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:14:01,176 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:14:02,122 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:02,172 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:14:03,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:03,173 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:14:04,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:05,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:06,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:06,215 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:07,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:08,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:09,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:10,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:11,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:11,216 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:12,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:13,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:14,133 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:15,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:16,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:16,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:17,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:18,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:19,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:20,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:21,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:21,217 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:22,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:23,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:24,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:25,134 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:26,135 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:26,218 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:27,146 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:28,139 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:29,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:29,247 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:14:30,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:31,142 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:31,248 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:32,143 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:32,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:14:32,327 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:14:32,327 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:14:32,329 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:14:33,198 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:14:33,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:34,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:35,200 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:14:35,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:36,366 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:36,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:37,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:38,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:39,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:40,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:41,367 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:41,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:42,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:43,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:44,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:45,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:46,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:46,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:47,397 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:48,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:49,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:50,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:51,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:51,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:52,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:53,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:54,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:55,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:56,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:14:56,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:57,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:58,402 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:14:59,256 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:14:59,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:00,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:01,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:02,259 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:02,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:03,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:03,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:15:03,480 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:15:03,481 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:15:03,482 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:15:04,240 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:15:04,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:05,241 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:15:05,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:06,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:07,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:07,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:08,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:09,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:10,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:11,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:12,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:12,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:13,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:14,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:15,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:16,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:17,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:17,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:18,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:19,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:20,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:21,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:22,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:22,522 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:23,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:24,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:25,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:26,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:27,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:27,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:28,410 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:29,253 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:15:29,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:30,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:31,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:32,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:33,262 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:33,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:33,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:15:33,544 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:15:33,544 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:15:33,549 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:15:34,277 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:15:34,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:35,278 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:15:35,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:36,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:37,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:38,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:38,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:39,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:40,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:41,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:42,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:43,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:43,584 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:44,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:45,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:46,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:47,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:48,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:48,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:49,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:50,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:51,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:52,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:53,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:53,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:54,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:55,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:56,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:57,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:58,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:15:58,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:15:59,258 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:15:59,414 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:00,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:01,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:02,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:03,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:04,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:04,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:05,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:06,041 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:16:06,042 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:16:06,043 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:16:06,048 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:16:06,310 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:16:06,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:07,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:08,443 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:09,313 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:16:09,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:10,083 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:10,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:11,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:12,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:13,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:14,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:15,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:15,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:16,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:17,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:18,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:19,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:20,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:20,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:21,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:22,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:23,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:24,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:25,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:25,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:26,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:27,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:28,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:29,261 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:16:29,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:30,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:30,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:31,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:32,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:33,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:34,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:35,264 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:35,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:36,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:37,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:38,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:39,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:40,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:40,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:41,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:41,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:16:41,863 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:16:41,864 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:16:41,867 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:16:42,342 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:16:42,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:43,343 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:16:43,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:44,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:45,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:45,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:46,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:47,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:48,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:49,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:50,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:50,905 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:51,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:52,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:53,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:54,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:55,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:55,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:16:56,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:57,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:58,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:16:59,266 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:16:59,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:00,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:01,268 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:01,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:02,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:03,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:04,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:05,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:06,269 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:06,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:07,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:08,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:09,636 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:10,635 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:11,275 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:11,634 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:12,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:13,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:14,119 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:17:14,121 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:17:14,121 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:17:14,126 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:17:14,383 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:17:14,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:15,384 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:17:15,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:16,638 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:17,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:17,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:18,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:19,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:20,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:21,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:22,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:22,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:23,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:24,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:25,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:26,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:27,161 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:27,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:28,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:29,270 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:17:29,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:30,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:31,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:32,272 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:32,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:33,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:34,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:35,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:36,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:37,273 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:37,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:38,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:39,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:40,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:41,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:42,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:42,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:43,650 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:44,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:45,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:45,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:17:45,863 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:17:45,864 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:17:45,870 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:17:46,419 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:17:46,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:47,420 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:17:47,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:47,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:48,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:49,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:50,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:51,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:52,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:52,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:53,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:54,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:55,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:56,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:57,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:57,904 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:17:58,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:17:59,274 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:17:59,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:00,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:01,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:02,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:03,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:03,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:04,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:05,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:06,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:07,649 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:08,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:08,664 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:09,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:10,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:11,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:12,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:13,286 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:13,661 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:14,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:18:14,436 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:18:14,436 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:18:14,437 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:18:14,451 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:18:14,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:15,453 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:18:15,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:16,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:17,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:18,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:18,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:19,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:20,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:21,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:22,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:23,478 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:23,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:24,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:25,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:26,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:27,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:28,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:28,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:29,279 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:18:29,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:30,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:31,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:32,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:33,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:34,281 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:34,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:35,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:36,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:37,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:38,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:39,282 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:39,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:40,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:41,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:42,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:43,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:44,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:44,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:45,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:46,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:18:46,280 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:18:46,280 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:18:46,281 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:18:46,484 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:18:46,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:47,485 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:18:47,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:48,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:49,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:49,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:50,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:51,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:52,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:53,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:54,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:18:54,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:55,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:56,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:57,854 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:58,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:18:59,282 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:18:59,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:00,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:00,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:01,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:02,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:03,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:04,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:05,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:05,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:06,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:07,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:08,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:09,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:10,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:10,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:11,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:12,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:13,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:14,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:15,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:15,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:15,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:19:15,965 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:19:15,966 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:19:15,967 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:19:16,508 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:19:16,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:17,509 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:19:17,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:18,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:19,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:20,861 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:21,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:21,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:22,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:23,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:24,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:25,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:26,006 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:26,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:27,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:28,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:29,286 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:19:29,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:30,862 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:31,288 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:31,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:32,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:33,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:34,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:35,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:36,289 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:36,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:37,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:38,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:39,863 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:40,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:41,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:41,872 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:42,873 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:43,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:44,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:45,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:46,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:19:46,179 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:19:46,180 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:19:46,181 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:19:46,536 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:19:46,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:47,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:47,537 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:19:47,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:48,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:49,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:50,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:51,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:52,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:52,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:53,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:54,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:55,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:56,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:57,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:19:58,108 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:59,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:19:59,290 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:20:00,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:01,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:02,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:02,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:03,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:04,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:05,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:06,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:07,003 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:07,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:08,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:09,004 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:10,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:11,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:12,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:12,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:13,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:14,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:15,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:15,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:20:15,239 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:20:15,239 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:20:15,241 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:20:15,562 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:20:16,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:17,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:17,563 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:20:18,012 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:18,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:19,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:20,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:21,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:22,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:23,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:23,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:24,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:25,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:26,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:27,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:28,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:28,280 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:29,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:29,293 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:20:30,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:31,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:32,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:33,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:33,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:34,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:35,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:36,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:37,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:38,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:38,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:39,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:40,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:41,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:42,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:43,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:43,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:44,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:44,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:20:44,438 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:20:44,439 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:20:44,441 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:20:44,585 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:20:45,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:45,587 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:20:46,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:47,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:48,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:48,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:49,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:50,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:51,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:52,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:53,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:53,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:54,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:55,020 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:56,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:57,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:58,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:58,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:20:59,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:20:59,297 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:21:00,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:01,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:02,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:03,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:04,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:04,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:05,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:06,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:07,226 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:08,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:09,234 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:09,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:10,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:11,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:12,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:13,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:13,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:21:13,825 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:21:13,825 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:21:13,826 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:21:14,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:14,612 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:21:14,865 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:15,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:15,613 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:21:16,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:17,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:18,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:19,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:19,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:20,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:21,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:22,237 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:23,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:24,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:24,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:25,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:26,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:27,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:28,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:29,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:29,302 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:21:30,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:30,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:31,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:32,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:33,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:34,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:35,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:35,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:36,239 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:37,248 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:38,243 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:39,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:40,244 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:40,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:41,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:42,245 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:43,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:21:43,021 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:21:43,022 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:21:43,023 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:21:43,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:43,644 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:21:44,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:45,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:45,646 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:21:46,060 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:46,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:47,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:48,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:49,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:50,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:51,061 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:51,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:52,426 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:53,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:54,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:55,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:56,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:21:56,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:57,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:58,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:21:59,305 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:21:59,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:00,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:01,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:01,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:02,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:03,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:04,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:05,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:06,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:06,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:07,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:08,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:09,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:10,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:11,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:11,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:11,824 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:22:11,825 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:22:11,826 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:22:11,827 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:22:12,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:12,670 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:22:13,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:13,670 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:22:14,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:15,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:16,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:16,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:17,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:18,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:19,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:20,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:21,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:21,866 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:22,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:23,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:24,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:25,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:26,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:26,867 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:27,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:28,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:29,309 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:22:29,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:30,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:31,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:32,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:32,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:33,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:34,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:35,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:36,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:37,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:37,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:38,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:39,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:40,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:41,486 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:42,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:42,490 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:42,906 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:22:42,907 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:22:42,907 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:22:42,908 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:22:43,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:43,698 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:22:43,698 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:22:44,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:45,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:46,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:47,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:47,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:48,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:49,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:50,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:51,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:52,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:52,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:53,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:54,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:55,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:56,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:57,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:57,948 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:22:58,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:22:59,313 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:22:59,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:00,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:01,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:02,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:03,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:03,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:04,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:05,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:06,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:07,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:08,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:08,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:09,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:10,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:11,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:12,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:13,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:13,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:14,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:15,493 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:16,177 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:23:16,179 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:23:16,179 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:23:16,181 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:23:16,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:16,723 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:23:17,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:17,724 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:23:18,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:19,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:19,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:20,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:21,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:22,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:23,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:24,219 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:24,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:25,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:26,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:27,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:28,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:29,220 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:29,317 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:23:29,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:30,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:31,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:32,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:33,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:34,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:34,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:35,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:36,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:37,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:38,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:39,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:39,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:40,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:41,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:42,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:43,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:44,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:44,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:45,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:45,669 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:23:45,670 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:23:45,671 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:23:45,673 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:23:45,752 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:23:46,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:47,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:47,754 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:23:48,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:49,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:49,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:50,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:51,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:52,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:53,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:54,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:54,711 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:23:55,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:56,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:57,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:58,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:23:59,321 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:23:59,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:00,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:00,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:01,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:02,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:03,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:04,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:05,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:05,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:06,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:07,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:08,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:09,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:10,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:10,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:11,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:12,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:13,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:14,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:15,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:15,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:16,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:17,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:18,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:24:18,533 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:24:18,534 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:24:18,535 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:24:18,785 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:24:18,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:19,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:19,786 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:24:20,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:20,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:21,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:22,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:23,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:24,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:25,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:25,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:26,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:27,675 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:28,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:29,325 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:24:29,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:30,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:31,327 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:31,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:32,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:33,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:34,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:35,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:36,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:36,676 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:37,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:38,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:39,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:40,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:41,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:41,677 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:42,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:43,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:44,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:45,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:46,334 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:46,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:47,680 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:48,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:48,789 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:24:48,791 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:24:48,791 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:24:48,792 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:24:48,816 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:24:49,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:49,818 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:24:50,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:51,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:51,830 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:52,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:53,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:54,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:55,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:56,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:56,831 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:24:57,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:58,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:24:59,329 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:24:59,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:00,681 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:01,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:02,331 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:02,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:03,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:04,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:05,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:06,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:07,332 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:07,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:08,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:09,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:10,682 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:11,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:12,333 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:12,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:13,683 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:14,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:15,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:16,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:17,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:17,692 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:18,689 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:19,643 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:25:19,644 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:25:19,644 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:25:19,646 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:25:19,845 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:25:20,042 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:20,786 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:21,786 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:21,847 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:25:22,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:22,786 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:23,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:24,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:25,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:26,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:27,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:27,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:28,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:29,333 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:25:29,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:30,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:31,787 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:32,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:33,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:33,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:34,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:35,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:36,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:37,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:38,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:38,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:39,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:40,811 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:41,812 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:42,814 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:43,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:43,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:44,823 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:45,820 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:46,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:47,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:48,342 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:48,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:25:48,622 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:25:48,622 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:25:48,624 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:25:48,873 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:25:49,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:49,874 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:25:50,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:51,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:52,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:53,030 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:53,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:54,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:55,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:56,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:57,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:58,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:58,662 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:25:59,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:25:59,337 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:26:00,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:01,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:02,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:03,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:04,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:04,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:05,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:06,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:07,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:08,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:09,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:09,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:10,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:11,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:12,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:13,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:14,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:14,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:15,043 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:16,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:16,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:26:16,884 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:26:16,885 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:26:16,886 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:26:16,896 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:26:17,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:17,898 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:26:18,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:19,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:19,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:20,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:21,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:22,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:23,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:24,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:24,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:25,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:26,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:27,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:28,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:29,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:29,341 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:26:30,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:30,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:31,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:32,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:33,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:34,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:35,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:35,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:36,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:37,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:38,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:39,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:40,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:40,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:41,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:42,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:43,299 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:44,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:45,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:45,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:46,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:46,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:26:46,308 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:26:46,308 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:26:46,309 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:26:46,925 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:26:47,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:47,927 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:26:48,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:49,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:50,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:51,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:51,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:52,301 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:53,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:54,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:55,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:56,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:56,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:26:57,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:58,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:59,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:26:59,345 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:27:00,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:01,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:02,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:02,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:03,302 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:04,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:05,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:06,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:07,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:07,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:08,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:09,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:10,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:11,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:12,313 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:12,355 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:13,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:14,310 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:15,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:16,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:27:16,279 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:27:16,279 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:27:16,281 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:27:16,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:16,952 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:27:17,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:17,953 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:27:18,319 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:18,684 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:19,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:20,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:21,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:22,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:23,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:23,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:24,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:25,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:26,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:27,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:28,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:28,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:29,349 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:27:29,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:30,685 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:31,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:32,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:33,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:33,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:34,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:35,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:36,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:37,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:38,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:38,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:39,686 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:40,693 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:41,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:42,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:43,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:43,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:44,696 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:45,690 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:46,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:27:46,383 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:27:46,384 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:27:46,385 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:27:46,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:46,975 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:27:47,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:47,976 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:27:48,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:48,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:49,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:50,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:51,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:52,792 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:53,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:53,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:54,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:55,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:56,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:57,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:58,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:27:58,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:27:59,353 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:27:59,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:00,793 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:01,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:02,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:03,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:04,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:04,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:05,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:06,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:07,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:08,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:09,357 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:09,794 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:10,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:11,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:12,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:13,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:14,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:14,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:15,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:16,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:28:16,548 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:28:16,549 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:28:16,550 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:28:16,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:17,006 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:28:17,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:18,007 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:28:18,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:19,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:19,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:20,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:21,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:22,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:23,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:24,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:24,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:25,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:26,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:27,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:28,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:29,357 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:28:29,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:30,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:30,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:31,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:32,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:33,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:34,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:35,360 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:35,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:36,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:37,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:38,957 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:39,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:40,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:40,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:41,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:42,958 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:43,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:44,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:45,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:45,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:46,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:47,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:48,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:49,672 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:28:49,674 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:28:49,674 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:28:49,676 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:28:50,038 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:28:50,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:50,714 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:51,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:52,040 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:28:52,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:53,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:54,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:55,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:55,715 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:28:56,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:57,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:58,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:59,077 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:28:59,361 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:29:00,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:01,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:01,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:02,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:03,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:04,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:05,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:06,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:06,364 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:07,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:08,078 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:09,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:10,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:11,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:11,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:12,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:13,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:14,079 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:15,089 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:16,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:16,371 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:17,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:18,084 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:19,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:20,085 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:20,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:29:20,600 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:29:20,600 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:29:20,602 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:29:21,063 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:29:21,130 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:21,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:22,064 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:29:22,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:23,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:24,128 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:25,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:26,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:26,642 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:27,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:28,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:29,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:29,366 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:29:30,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:31,129 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:32,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:32,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:33,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:34,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:35,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:36,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:37,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:37,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:38,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:39,204 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:40,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:41,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:42,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:42,370 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:43,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:44,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:45,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:46,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:47,214 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:47,378 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:48,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:49,207 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:50,033 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:29:50,034 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:29:50,035 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:29:50,036 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:29:50,086 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:29:50,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:51,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:52,088 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:29:52,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:53,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:53,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:54,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:55,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:56,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:57,444 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:58,076 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:29:58,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:29:59,369 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:29:59,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:00,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:01,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:02,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:03,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:03,448 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:04,445 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:05,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:06,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:07,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:08,372 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:08,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:09,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:10,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:11,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:12,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:13,373 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:13,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:14,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:15,446 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:16,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:17,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:18,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:18,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:19,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:20,452 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:21,449 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:22,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:30:22,389 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:30:22,389 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:30:22,391 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:30:22,796 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:23,111 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:30:23,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:23,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:24,112 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:30:24,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:25,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:26,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:27,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:28,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:28,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:29,373 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:30:29,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:30,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:31,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:32,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:33,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:34,375 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:34,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:35,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:36,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:37,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:38,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:39,376 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:39,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:40,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:41,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:42,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:43,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:44,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:44,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:45,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:46,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:47,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:48,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:49,377 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:49,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:50,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:51,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:52,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:53,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:54,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:30:54,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:55,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:56,276 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:30:56,277 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:30:56,278 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:30:56,279 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:30:56,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:57,142 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:30:57,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:58,143 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:30:58,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:30:59,377 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:30:59,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:00,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:00,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:01,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:02,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:03,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:04,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:05,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:05,678 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:06,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:07,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:08,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:09,679 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:10,380 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:10,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:11,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:12,741 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:13,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:14,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:15,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:15,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:16,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:17,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:18,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:19,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:20,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:20,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:21,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:22,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:23,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:24,746 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:25,386 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:25,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:26,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:27,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:28,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:31:28,229 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:31:28,229 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:31:28,231 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:31:28,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:29,169 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:31:29,380 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:31:29,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:30,171 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:31:30,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:31,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:31,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:32,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:33,750 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:34,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:35,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:36,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:36,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:37,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:38,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:39,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:40,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:41,383 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:41,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:42,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:43,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:44,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:45,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:46,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:46,751 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:47,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:48,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:49,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:50,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:51,384 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:51,752 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:52,755 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:53,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:54,763 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:55,759 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:56,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:31:56,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:57,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:57,983 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:31:57,985 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:31:57,985 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:31:57,987 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:31:58,194 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:31:58,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:31:59,385 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:31:59,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:00,196 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:32:00,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:01,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:02,387 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:02,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:03,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:04,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:05,764 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:06,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:07,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:07,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:08,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:09,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:10,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:11,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:12,388 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:12,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:13,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:14,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:15,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:16,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:17,389 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:17,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:18,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:19,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:20,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:21,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:22,390 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:22,891 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:23,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:24,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:25,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:26,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:27,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:27,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:28,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:29,388 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:32:29,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:32:29,462 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:32:29,462 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:32:29,463 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:32:29,898 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:30,224 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:32:30,225 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:32:30,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:31,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:32,502 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:32,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:33,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:34,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:35,900 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:36,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:37,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:37,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:38,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:39,901 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:41,037 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:41,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:42,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:42,940 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:43,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:44,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:45,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:46,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:47,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:47,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:48,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:49,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:50,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:51,941 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:52,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:52,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:53,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:54,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:55,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:56,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:57,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:32:57,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:58,265 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:32:58,266 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:32:58,266 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:32:58,267 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:32:58,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:32:59,255 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:32:59,392 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:32:59,951 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:00,256 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:33:00,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:01,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:02,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:03,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:03,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:04,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:05,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:06,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:07,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:08,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:08,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:09,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:10,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:11,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:12,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:13,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:13,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:14,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:15,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:16,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:17,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:18,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:18,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:19,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:20,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:21,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:22,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:23,406 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:23,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:24,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:25,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:26,512 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:33:26,514 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:33:26,515 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:33:26,516 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:33:26,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:27,281 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:33:27,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:28,281 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:33:28,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:28,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:29,396 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:33:29,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:30,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:31,954 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:32,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:33,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:34,399 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:34,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:35,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:36,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:37,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:38,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:39,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:39,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:40,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:41,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:42,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:43,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:44,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:44,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:45,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:46,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:47,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:48,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:49,401 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:49,956 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:50,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:51,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:52,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:53,968 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:54,405 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:33:54,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:55,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:56,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:33:56,341 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:33:56,341 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:33:56,342 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:33:56,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:57,307 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:33:57,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:58,308 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:33:58,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:33:59,400 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:33:59,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:00,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:00,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:01,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:02,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:03,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:04,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:05,403 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:05,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:06,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:07,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:08,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:09,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:10,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:10,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:11,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:12,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:13,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:14,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:15,404 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:15,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:16,965 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:17,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:18,966 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:19,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:20,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:20,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:21,973 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:22,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:23,914 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:34:23,915 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:34:23,915 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:34:23,917 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:34:24,326 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:24,343 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:34:24,344 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:34:25,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:25,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:26,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:27,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:28,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:29,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:29,404 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:34:30,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:31,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:31,407 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:32,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:33,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:34,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:35,056 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:36,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:36,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:37,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:38,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:39,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:40,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:41,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:41,408 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:42,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:43,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:44,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:45,057 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:46,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:46,409 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:47,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:48,058 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:49,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:50,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:51,062 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:51,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:52,065 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:53,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:54,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:54,544 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:34:54,546 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:34:54,547 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:34:54,548 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:34:55,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:55,377 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:34:56,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:56,377 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:34:56,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:34:57,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:58,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:59,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:34:59,409 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:35:00,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:01,066 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:02,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:02,411 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:03,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:04,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:05,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:06,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:07,067 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:07,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:08,202 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:09,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:10,199 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:11,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:12,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:12,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:13,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:14,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:15,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:16,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:17,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:17,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:18,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:19,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:20,205 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:21,206 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:22,210 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:22,418 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:23,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:24,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:25,209 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:25,890 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:35:25,892 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:35:25,892 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:35:25,894 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:35:26,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:26,411 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:35:26,412 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:35:27,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:27,932 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:28,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:29,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:29,413 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:35:30,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:31,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:32,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:33,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:33,415 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:34,295 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:35,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:36,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:37,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:38,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:38,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:39,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:40,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:41,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:42,296 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:43,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:43,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:44,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:45,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:46,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:47,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:48,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:48,417 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:49,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:50,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:51,297 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:52,303 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:53,300 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:53,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:54,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:55,309 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:56,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:57,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:57,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:35:57,566 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:35:57,566 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:35:57,568 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:35:58,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:58,449 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:35:58,449 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:35:58,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:35:59,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:35:59,417 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:36:00,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:01,304 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:02,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:03,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:04,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:04,419 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:05,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:06,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:07,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:08,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:09,305 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:09,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:10,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:11,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:12,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:13,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:14,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:14,420 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:15,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:16,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:17,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:18,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:19,306 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:19,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:20,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:21,307 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:22,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:23,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:24,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:24,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:25,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:26,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:26,416 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:36:26,417 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:36:26,418 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:36:26,419 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:36:26,478 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:36:27,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:28,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:28,479 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:36:29,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:29,421 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:36:30,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:30,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:31,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:32,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:33,314 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:34,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:35,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:35,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:36,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:37,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:38,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:39,315 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:40,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:40,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:41,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:42,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:43,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:44,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:45,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:45,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:46,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:47,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:48,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:49,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:50,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:50,425 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:51,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:52,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:53,356 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:54,347 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:55,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:55,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:36:56,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:57,348 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:58,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:58,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:36:58,912 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:36:58,912 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:36:58,914 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:36:59,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:36:59,425 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:36:59,507 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:37:00,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:00,508 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:37:01,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:01,427 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:02,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:03,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:04,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:05,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:06,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:06,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:07,349 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:08,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:09,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:10,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:11,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:11,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:12,350 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:13,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:14,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:15,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:16,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:16,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:17,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:18,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:19,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:20,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:21,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:21,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:22,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:23,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:24,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:25,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:26,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:26,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:27,365 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:28,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:29,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:29,430 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:37:30,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:30,726 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:37:30,727 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:37:30,727 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:37:30,728 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:37:31,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:31,535 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:37:31,767 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:32,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:32,536 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:37:33,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:34,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:35,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:36,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:36,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:37,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:38,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:39,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:40,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:41,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:41,768 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:42,363 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:43,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:44,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:45,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:46,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:46,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:47,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:48,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:49,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:50,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:51,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:51,769 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:52,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:53,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:54,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:55,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:56,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:56,777 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:37:57,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:58,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:37:59,436 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:37:59,442 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:00,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:01,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:38:01,279 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:38:01,280 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:38:01,281 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:38:01,565 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:38:02,320 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:02,499 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:03,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:03,567 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:38:04,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:05,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:06,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:07,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:07,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:08,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:09,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:10,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:11,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:12,321 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:12,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:13,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:14,498 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:15,655 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:16,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:17,322 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:17,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:18,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:19,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:20,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:21,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:22,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:22,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:23,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:24,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:25,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:26,562 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:27,324 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:27,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:28,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:29,442 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:38:29,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:30,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:31,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:32,447 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:32,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:33,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:34,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:38:34,004 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:38:34,004 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:38:34,006 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:38:34,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:34,593 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:38:35,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:35,594 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:38:36,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:37,572 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:38,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:38,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:39,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:40,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:41,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:42,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:43,045 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:43,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:44,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:45,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:46,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:47,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:48,046 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:48,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:49,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:50,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:51,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:52,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:53,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:53,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:54,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:55,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:56,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:57,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:58,047 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:38:58,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:38:59,442 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:38:59,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:00,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:01,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:02,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:03,450 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:03,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:04,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:05,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:05,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:39:05,604 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:39:05,605 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:39:05,606 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:39:05,627 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:39:06,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:07,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:07,629 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:39:08,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:08,645 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:09,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:10,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:11,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:12,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:13,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:13,646 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:14,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:15,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:16,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:17,586 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:18,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:18,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:19,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:20,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:21,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:22,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:23,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:23,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:24,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:25,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:26,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:27,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:28,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:28,648 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:29,446 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:39:29,588 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:30,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:31,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:32,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:33,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:34,457 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:34,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:35,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:35,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:39:35,882 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:39:35,883 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:39:35,885 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:39:36,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:36,661 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:39:37,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:37,662 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:39:38,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:39,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:39,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:40,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:41,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:42,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:43,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:44,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:44,924 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:45,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:46,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:47,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:48,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:49,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:49,925 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:50,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:51,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:52,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:53,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:54,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:54,926 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:39:55,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:56,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:57,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:58,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:39:59,451 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:39:59,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:00,460 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:00,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:01,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:02,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:03,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:04,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:05,463 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:05,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:05,673 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:40:05,674 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:40:05,675 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:40:05,676 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:40:05,685 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:40:06,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:07,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:07,687 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:40:08,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:09,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:10,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:10,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:11,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:12,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:13,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:14,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:15,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:15,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:16,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:17,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:18,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:19,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:20,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:20,717 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:21,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:22,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:23,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:24,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:25,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:25,718 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:26,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:27,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:28,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:29,454 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:40:29,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:30,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:31,466 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:31,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:32,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:33,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:34,621 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:35,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:36,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:36,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:36,700 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:40:36,701 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:40:36,702 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:40:36,703 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:40:36,710 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:40:37,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:38,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:39,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:39,711 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:40:40,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:41,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:41,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:42,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:43,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:44,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:45,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:46,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:46,742 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:47,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:48,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:49,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:50,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:51,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:51,743 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:52,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:53,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:54,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:55,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:56,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:56,744 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:40:57,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:58,701 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:40:59,458 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:40:59,702 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:00,702 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:01,702 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:02,461 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:02,716 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:03,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:04,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:05,712 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:06,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:07,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:07,709 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:08,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:41:08,530 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:41:08,531 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:41:08,532 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:41:08,741 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:41:08,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:09,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:10,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:11,743 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:41:11,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:12,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:12,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:13,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:14,937 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:15,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:16,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:17,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:17,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:18,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:19,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:20,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:21,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:22,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:22,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:23,938 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:24,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:25,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:26,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:27,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:27,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:28,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:29,462 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:41:29,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:30,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:31,939 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:32,952 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:33,469 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:33,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:34,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:35,949 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:36,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:37,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:38,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:38,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:41:38,509 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:41:38,510 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:41:38,511 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:41:38,774 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:41:38,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:39,774 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:41:39,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:40,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:41,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:42,945 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:43,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:43,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:44,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:45,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:46,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:47,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:48,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:48,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:49,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:50,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:51,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:52,946 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:53,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:53,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:54,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:55,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:56,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:57,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:58,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:41:58,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:41:59,466 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:41:59,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:00,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:01,947 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:02,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:03,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:04,474 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:04,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:05,955 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:06,964 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:07,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:08,929 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:42:08,931 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:42:08,931 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:42:08,933 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:42:09,341 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:09,805 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:42:09,970 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:10,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:11,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:11,806 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:42:12,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:13,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:14,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:14,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:15,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:16,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:17,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:18,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:19,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:19,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:20,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:21,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:22,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:23,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:24,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:24,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:25,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:26,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:27,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:28,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:29,344 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:29,470 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:42:30,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:30,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:31,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:32,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:33,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:34,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:35,345 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:35,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:36,346 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:37,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:38,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:39,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:40,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:40,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:41,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:41,369 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:42:41,371 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:42:41,371 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:42:41,373 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:42:41,830 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:42:42,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:43,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:43,831 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:42:44,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:45,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:46,352 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:46,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:47,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:48,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:49,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:50,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:51,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:51,412 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:52,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:53,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:54,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:55,353 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:56,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:56,413 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:42:57,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:58,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:59,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:42:59,475 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:43:00,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:01,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:01,477 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:02,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:03,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:04,354 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:05,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:06,358 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:06,484 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:07,359 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:08,361 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:09,368 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:10,362 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:11,251 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:43:11,252 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:43:11,253 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:43:11,254 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:43:11,665 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:11,860 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:43:12,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:12,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:13,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:13,861 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:43:14,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:15,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:16,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:17,293 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:17,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:18,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:19,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:20,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:21,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:22,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:22,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:23,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:24,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:25,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:26,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:27,294 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:27,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:28,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:29,477 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:43:29,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:30,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:31,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:32,479 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:32,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:33,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:34,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:35,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:36,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:37,489 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:37,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:38,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:39,541 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:40,540 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:41,379 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:43:41,381 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:43:41,382 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:43:41,383 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:43:41,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:41,886 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:43:42,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:43,421 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:43,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:43,888 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:43:44,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:45,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:46,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:47,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:48,422 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:48,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:49,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:50,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:51,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:52,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:53,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:53,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:54,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:55,930 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:56,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:57,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:58,423 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:43:58,843 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:43:59,481 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:43:59,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:00,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:01,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:02,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:03,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:03,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:04,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:05,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:06,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:07,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:08,488 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:08,856 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:09,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:10,847 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:11,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:12,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:13,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:13,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:14,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:44:14,397 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:44:14,397 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:44:14,398 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:44:14,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:14,915 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:44:15,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:15,916 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:44:16,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:17,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:18,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:19,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:19,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:20,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:21,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:22,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:23,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:24,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:24,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:25,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:26,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:27,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:28,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:29,440 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:29,485 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:44:29,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:30,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:31,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:32,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:33,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:34,487 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:34,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:35,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:36,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:37,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:38,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:39,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:39,855 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:40,858 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:41,860 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:42,859 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:43,561 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:44:43,562 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:44:43,562 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:44:43,563 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:44:43,944 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:44:43,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:44,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:44,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:45,946 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:44:45,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:46,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:47,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:48,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:49,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:49,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:50,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:51,961 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:52,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:53,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:54,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:44:54,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:55,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:56,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:57,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:58,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:44:59,489 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:44:59,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:00,491 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:00,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:01,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:02,962 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:03,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:04,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:05,492 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:05,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:06,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:07,963 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:08,977 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:09,967 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:10,495 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:10,969 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:11,971 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:12,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:13,972 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:14,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:45:14,617 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:45:14,617 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:45:14,618 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:45:14,972 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:45:15,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:15,656 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:15,973 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:45:16,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:17,001 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:18,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:19,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:20,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:20,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:21,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:22,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:23,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:24,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:25,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:25,657 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:26,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:27,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:28,002 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:29,009 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:29,494 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:45:30,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:31,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:31,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:32,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:33,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:34,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:35,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:36,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:36,497 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:37,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:38,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:39,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:40,008 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:41,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:41,503 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:42,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:43,011 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:44,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:45,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:46,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:46,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:46,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:45:46,578 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:45:46,578 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:45:46,580 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:45:47,003 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:45:47,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:48,004 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:45:48,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:49,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:50,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:51,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:51,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:52,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:53,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:54,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:55,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:56,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:56,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:45:57,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:58,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:59,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:45:59,497 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:46:00,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:01,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:02,017 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:02,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:03,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:04,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:05,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:06,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:07,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:07,500 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:08,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:09,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:10,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:11,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:12,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:12,513 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:13,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:13,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:46:13,730 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:46:13,731 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:46:13,732 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:46:14,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:14,032 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:46:15,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:16,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:16,034 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:46:17,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:17,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:18,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:19,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:20,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:21,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:22,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:22,772 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:23,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:24,026 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:25,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:26,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:27,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:27,773 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:28,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:29,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:29,501 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:46:30,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:31,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:32,048 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:33,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:33,504 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:34,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:35,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:36,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:37,049 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:39,480 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:39,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:40,048 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:46:44,481 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:45,844 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:46:46,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:46:46,284 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:47,163 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:48,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:49,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:53,980 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:46:54,482 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:54,810 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:55,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:46:59,483 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:46:59,505 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:47:02,136 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:47:02,451 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:03,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:04,508 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:04,781 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:09,509 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:10,400 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:47:10,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:47:11,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:12,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:14,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:19,510 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:19,576 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:47:19,818 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:47:19,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:21,749 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:22,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:24,511 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:29,468 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:47:29,509 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:47:29,516 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:30,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:47:33,833 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:33,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:34,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:34,874 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:35,834 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:36,208 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:47:36,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:37,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:38,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:39,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:39,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:40,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:41,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:42,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:43,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:44,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:44,875 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:45,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:46,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:47,835 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:48,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:49,836 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:49,876 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:50,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:51,848 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:52,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:53,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:54,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:54,877 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:47:55,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:56,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:57,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:58,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:47:59,514 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:47:59,849 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:00,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:00,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:01,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:02,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:03,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:04,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:05,517 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:05,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:06,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:07,850 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:08,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:09,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:10,518 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:10,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:11,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:12,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:13,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:14,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:15,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:15,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:16,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:17,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:18,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:19,852 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:20,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:20,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:21,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:22,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:23,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:24,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:25,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:25,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:26,853 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:29,518 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:48:31,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:36,521 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:41,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:46,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:47,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:48:47,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:48:47,465 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:48,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:49,428 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:50,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:51,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:51,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:52,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:53,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:54,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:55,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:56,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:56,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:48:57,429 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:58,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:59,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:48:59,522 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:49:00,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:01,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:02,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:02,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:03,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:04,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:05,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:06,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:07,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:07,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:08,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:09,430 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:10,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:11,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:12,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:12,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:13,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:14,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:15,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:16,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:17,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:17,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:18,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:19,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:20,431 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:21,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:22,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:22,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:23,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:24,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:25,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:26,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:27,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:27,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:28,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:29,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:29,526 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:49:30,432 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:31,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:32,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:32,528 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:33,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:34,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:35,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:36,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:37,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:37,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:38,433 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:39,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:40,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:41,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:42,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:42,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:43,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:44,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:45,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:46,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:47,434 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:47,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:48,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:49,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:50,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:51,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:52,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:52,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:53,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:54,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:55,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:56,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:57,435 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:57,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:49:58,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:59,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:49:59,530 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:50:00,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:01,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:02,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:02,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:03,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:04,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:05,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:06,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:07,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:07,533 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:08,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:09,436 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:10,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:11,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:12,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:12,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:13,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:14,437 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:15,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:16,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:17,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:17,534 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:18,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:19,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:20,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:21,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:22,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:22,535 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:23,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:24,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:25,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:26,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:27,438 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:27,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:28,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:29,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:29,534 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:50:30,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:50:32,536 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:37,537 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:42,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:47,538 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:52,539 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:57,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:50:59,545 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:51:02,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:07,548 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:11,276 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 15:51:11,316 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:12,276 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:12,549 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:13,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:14,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:15,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:16,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:16,466 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:51:17,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:18,223 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:18,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:18,467 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:51:19,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:20,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:21,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:22,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:23,224 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:23,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:24,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:25,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:26,277 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:27,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:28,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:28,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:29,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:29,539 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:51:30,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:31,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:32,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:33,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:33,542 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:34,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:35,278 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:36,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:37,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:38,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:38,543 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:39,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:40,279 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:41,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:42,285 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:43,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:43,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:44,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:45,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:46,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:46,351 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:51:46,352 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:51:46,352 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:51:46,355 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:51:46,489 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:51:47,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:48,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:48,490 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:51:49,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:49,393 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:50,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:51,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:52,291 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:53,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:54,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:54,394 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:55,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:56,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:57,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:58,292 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:59,395 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:51:59,424 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:51:59,544 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:52:00,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:01,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:02,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:03,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:04,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:04,546 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:05,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:06,335 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:07,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:08,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:09,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:09,547 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:10,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:11,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:12,343 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:13,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:14,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:14,552 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:15,336 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:16,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:17,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:17,472 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:52:17,473 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:52:17,474 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:52:17,479 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:52:17,511 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:52:18,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:19,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:20,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:20,513 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:52:20,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:21,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:22,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:23,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:24,338 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:25,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:25,514 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:26,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:27,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:28,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:29,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:29,548 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:52:30,339 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:30,550 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:31,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:32,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:33,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:34,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:35,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:35,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:36,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:37,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:38,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:39,519 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:40,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:40,551 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:41,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:42,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:43,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:44,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:45,520 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:45,553 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:46,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:47,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:48,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:49,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:50,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:50,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:50,851 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:52:50,853 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:52:50,853 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:52:50,858 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:52:51,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:51,542 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:52:52,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:52,543 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:52:53,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:54,524 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:55,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:55,892 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:52:56,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:57,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:58,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:59,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:52:59,551 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:53:00,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:01,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:01,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:02,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:03,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:04,525 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:05,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:06,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:06,554 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:07,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:08,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:09,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:10,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:11,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:11,555 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:12,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:13,526 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:14,527 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:15,531 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:16,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:16,560 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:17,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:18,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:19,532 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:20,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:20,729 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:53:20,730 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:53:20,731 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:53:20,732 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:53:21,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:21,576 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:53:21,770 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:22,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:22,577 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:53:23,529 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:24,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:25,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:26,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:26,771 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:27,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:28,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:29,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:29,555 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:53:30,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:31,530 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:32,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:32,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:33,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:34,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:35,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:36,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:37,558 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:37,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:38,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:39,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:40,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:41,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:42,559 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:42,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:43,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:44,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:45,596 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:46,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:47,567 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:47,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:48,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:49,118 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:53:49,119 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:53:49,119 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:53:49,120 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:53:49,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:49,602 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:53:50,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:50,603 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:53:51,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:52,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:53,159 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:53,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:54,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:55,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:56,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:57,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:58,160 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:53:58,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:53:59,560 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:53:59,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:00,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:01,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:02,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:03,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:03,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:04,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:05,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:06,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:07,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:08,563 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:08,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:09,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:10,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:11,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:12,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:13,564 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:13,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:14,600 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:15,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:16,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:17,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:18,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:18,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:19,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:20,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:20,724 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:54:20,725 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:54:20,726 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:54:20,727 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:54:21,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:21,630 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:54:22,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:22,630 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:54:23,604 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:23,765 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:24,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:25,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:26,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:27,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:28,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:28,766 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:29,562 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:54:29,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:30,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:31,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:32,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:33,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:34,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:34,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:35,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:36,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:37,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:38,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:39,565 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:39,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:40,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:41,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:42,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:43,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:44,566 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:44,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:45,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:46,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:47,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:48,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:49,571 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:49,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:50,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:51,113 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:54:51,115 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:54:51,115 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:54:51,117 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:54:51,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:51,657 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:54:52,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:52,658 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:54:53,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:54,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:55,155 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:54:55,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:56,612 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:57,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:58,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:54:59,566 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:54:59,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:00,568 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:00,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:01,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:02,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:03,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:04,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:05,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:05,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:06,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:07,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:08,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:09,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:10,569 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:10,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:11,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:12,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:13,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:14,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:15,570 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:15,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:16,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:17,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:18,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:19,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:20,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:20,620 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:21,473 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:55:21,475 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:55:21,476 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:55:21,477 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:55:21,687 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:55:21,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:22,688 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:55:22,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:23,880 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:24,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:25,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:26,515 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:26,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:27,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:28,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:29,571 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:55:29,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:30,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:31,573 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:31,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:32,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:33,881 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:34,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:35,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:36,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:36,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:37,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:38,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:39,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:40,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:41,574 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:41,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:42,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:43,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:44,882 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:45,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:46,575 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:46,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:47,883 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:48,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:49,887 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:50,889 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:51,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:51,893 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:52,897 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:53,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:54,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:54,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:55:54,913 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:55:54,914 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:55:54,915 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:55:55,715 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:55:55,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:56,716 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:55:56,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:56,953 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:55:57,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:58,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:55:59,574 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:55:59,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:00,894 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:01,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:02,577 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:02,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:03,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:04,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:05,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:06,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:07,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:07,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:08,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:09,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:10,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:11,895 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:12,578 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:12,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:13,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:14,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:15,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:16,896 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:17,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:17,908 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:18,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:19,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:20,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:21,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:22,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:22,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:23,454 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:56:23,455 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:56:23,456 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:56:23,457 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:56:23,743 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:56:23,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:24,744 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:56:24,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:25,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:26,902 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:27,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:28,496 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:28,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:29,579 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:56:29,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:30,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:31,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:32,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:33,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:33,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:34,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:35,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:36,903 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:37,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:38,582 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:38,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:39,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:40,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:41,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:42,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:43,583 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:43,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:44,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:45,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:46,911 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:47,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:48,592 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:48,912 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:49,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:50,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:51,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:52,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:53,340 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:56:53,341 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:56:53,342 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:56:53,343 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:56:53,775 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:56:53,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:54,381 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:54,776 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:56:54,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:55,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:56,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:57,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:58,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:56:59,382 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:56:59,582 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:56:59,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:00,916 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:01,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:02,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:03,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:04,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:04,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:05,917 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:07,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:08,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:09,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:09,585 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:10,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:11,013 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:12,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:13,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:14,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:14,587 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:15,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:16,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:17,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:18,014 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:19,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:19,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:20,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:21,018 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:22,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:23,019 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:24,015 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:24,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:25,016 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:25,070 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:57:25,071 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:57:25,071 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:57:25,073 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:57:25,802 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:57:26,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:26,803 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:57:27,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:28,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:29,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:29,586 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:57:30,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:30,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:31,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:32,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:33,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:34,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:35,022 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:35,589 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:36,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:37,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:38,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:39,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:40,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:40,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:41,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:42,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:43,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:44,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:45,023 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:45,590 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:46,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:47,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:48,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:49,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:50,024 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:50,591 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:51,025 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:52,035 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:53,029 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:54,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:55,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:55,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:57:56,040 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:57,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:57,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:57:57,082 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:57:57,083 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:57:57,084 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:57:57,827 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:57:58,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:58,828 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:57:59,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:57:59,590 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:58:00,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:01,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:01,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:02,031 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:03,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:04,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:05,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:06,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:06,593 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:07,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:08,032 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:09,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:10,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:11,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:11,594 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:12,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:13,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:14,191 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:15,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:16,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:16,595 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:17,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:18,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:19,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:20,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:21,192 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:21,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:22,193 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:23,200 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:24,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:25,194 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:26,197 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:26,605 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:27,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:27,238 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:58:27,240 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:58:27,240 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:58:27,242 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:58:27,856 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:58:28,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:28,857 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:58:29,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:29,595 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:58:30,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:31,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:32,195 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:32,597 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:33,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:34,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:35,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:36,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:37,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:37,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:38,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:39,196 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:40,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:41,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:42,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:42,598 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:43,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:44,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:45,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:46,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:47,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:47,599 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:48,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:49,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:50,221 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:51,222 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:52,225 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:52,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:53,233 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:54,228 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:55,235 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:56,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:57,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:57,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:58:57,748 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:58:57,750 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:58:57,750 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:58:57,752 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:58:57,882 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:58:58,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:58,883 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:58:59,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:58:59,599 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:59:00,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:01,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:02,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:03,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:03,601 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:04,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:05,231 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:06,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:07,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:08,232 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:08,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:09,398 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:10,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:11,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:12,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:13,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:13,602 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:14,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:15,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:16,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:17,311 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:18,312 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:18,603 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:19,323 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:20,317 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:21,328 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:22,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:23,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:23,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:24,325 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:25,211 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:59:25,212 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:59:25,213 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:59:25,214 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:59:25,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:25,906 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:59:26,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:26,906 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:59:27,616 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:28,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:29,252 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:29,603 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:59:29,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:30,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:31,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:32,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:33,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:34,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:34,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:35,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:36,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:37,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:38,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:39,606 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:39,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:40,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:41,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:42,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:43,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:44,607 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:44,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:45,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:46,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:47,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:48,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:49,608 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:49,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:50,619 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:51,623 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:52,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:53,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:54,611 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 15:59:54,626 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:55,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:56,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:56,864 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 15:59:56,866 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 15:59:56,867 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 15:59:56,868 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 15:59:56,933 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 15:59:57,625 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:58,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 15:59:58,934 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 15:59:59,606 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 15:59:59,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:00,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:00,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:01,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:02,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:03,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:04,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:05,609 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:05,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:06,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:07,627 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:08,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:09,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:10,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:10,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:11,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:12,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:13,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:14,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:15,610 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:15,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:16,628 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:17,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:18,629 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:19,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:20,618 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:20,630 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:21,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:22,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:23,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:24,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:25,039 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 16:00:25,041 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 16:00:25,041 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 16:00:25,043 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 16:00:25,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:25,961 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 16:00:26,080 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:26,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:26,961 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 16:00:27,632 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:28,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:29,610 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 16:00:29,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:30,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:31,613 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:31,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:32,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:33,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:34,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:35,633 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:36,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:36,736 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:37,639 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:38,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:39,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:40,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:41,614 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:41,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:42,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:43,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:44,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:45,640 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:46,615 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:46,641 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:47,647 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:48,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:49,652 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:50,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:51,624 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:51,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:52,651 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:53,396 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 16:00:53,397 DEBUG SenderThread:3286844 [sender.py:send():379] send: history 2025-01-04 16:00:53,398 DEBUG SenderThread:3286844 [sender.py:send_request():406] send_request: summary_record 2025-01-04 16:00:53,399 INFO SenderThread:3286844 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 16:00:53,803 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:53,994 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/wandb-summary.json 2025-01-04 16:00:54,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:54,995 INFO Thread-12 :3286844 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250104_084255-r3b53lzn/files/output.log 2025-01-04 16:00:55,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:56,798 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:57,439 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:00:57,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:58,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:00:59,614 DEBUG SenderThread:3286844 [sender.py:send():379] send: stats 2025-01-04 16:00:59,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:01:00,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:01:01,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:01:02,617 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 16:01:02,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:01:03,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:01:04,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:01:05,799 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 16:01:06,800 DEBUG HandlerThread:3286844 [handler.py:handle_request():158] handle_request: internal_messages